Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

SSH "Permission denied (publickey)": what the server log says and what to fix

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

user@host: Permission denied (publickey). means the server accepted the connection and then rejected every public key the client offered. The client is told only which methods the server would still accept, in brackets. The reason is written to the server's log, so read the server log before you change a key or a setting.

Server-side causes include file or directory ownership and modes that sshd refuses, a key in a file sshd does not read, a user that sshd's access settings exclude, and a key type that the server no longer accepts.

What the error means

In the OpenSSH client source the message is built as "%s@%s: Permission denied (%s).", filled with the user name, the host and the list of authentication methods the server still allows. If that list is only publickey, the server accepts nothing but keys, and none worked. If it also lists password or keyboard-interactive, other methods remain.

On the server, the matching line is written when a method fails. In the source its form is %s %s%s%s for %s%.100s from %.200s port %d ssh2%s%s, which reads in practice as Failed publickey for deploy from 203.0.113.50 port 51234 ssh2, with the words invalid user added before the name when the account does not exist. A successful login uses Accepted in place of Failed, followed by details of the key.

Between those two lines, the server may log the actual reason at normal log level. Messages in the OpenSSH source include:

  • Authentication refused: bad ownership or modes for directory <path> and ... for file <path>
  • User <name> from <host> not allowed because not listed in AllowUsers
  • User <name> from <host> not allowed because listed in DenyUsers
  • User <name> not allowed because account is locked
  • signature algorithm <name> not in PubkeyAcceptedAlgorithms
  • User '<name>' authorized keys '<file>' is not a regular file

A missing authorized_keys file is not one of them. In the source it is logged only at debug level, so on a default log level a missing file leaves no hint.

Likely causes

Cause What you see How to confirm
Home directory, ~/.ssh or authorized_keys writable by group or others, or owned by someone else Authentication refused: bad ownership or modes for ... ls -ld on the home directory, .ssh and the file
Public key not in the file sshd reads Failed publickey with no other line The key text against the file; sshd -T shows the AuthorizedKeysFile setting
The account is excluded by AllowUsers, DenyUsers, AllowGroups or DenyGroups not allowed because ... sshd -T -C user=<name>,host=<host>,addr=<ip>
Locked account or an unusable shell not allowed because account is locked or a shell that does not exist or is not executable The log line
Key algorithm not in PubkeyAcceptedAlgorithms signature algorithm ... not in PubkeyAcceptedAlgorithms sshd -T and the key type in use
Public key login off, or AuthenticationMethods needs more than a key Failures even with a correct key sshd -T shows PubkeyAuthentication and AuthenticationMethods
The client offers a different key from the one installed Server log shows a failure; client shows other identities offered ssh -v on the client

Prove the cause before changing anything

On the client, run the connection with verbose output. The ssh manual says -v "Causes ssh to print debugging messages about its progress. This is helpful in debugging connection, authentication, and configuration problems", and repeating it raises the level to 3.

ssh -v deploy@203.0.113.10

Note which keys it offers and which method list the server returns. To see the public key behind a private key, ssh-keygen -y -f ~/.ssh/id_ed25519 prints it, so you can compare it with the line in the server's file.

On the server, read the journal for the SSH unit. Ubuntu's documentation shows sudo journalctl -fu ssh.service; the unit is named ssh.service there, and may be sshd.service on other distributions, and some systems write to a separate authentication log file.

sudo journalctl -u ssh.service
sudo sshd -T -C user=deploy,host=client.example.com,addr=198.51.100.7
ls -ld /home/deploy /home/deploy/.ssh /home/deploy/.ssh/authorized_keys

sshd -T is documented as "Extended test mode. Check the validity of the configuration file, output the effective configuration to stdout and then exit." With -C it applies Match blocks for the connection you describe, using keywords addr, user, host, laddr, lport and rdomain. Read the output for the AllowUsers, DenyUsers, AllowGroups, AuthorizedKeysFile, PubkeyAuthentication, PubkeyAcceptedAlgorithms and AuthenticationMethods settings.

The rule behind the ownership message is in the source. With StrictModes on, the file and every directory above it up to the home directory must be owned by the user or by root, and none may be group- or world-writable. The test is (st_mode & 022) != 0. The sshd manual puts it in words: "If this file, the ~/.ssh directory, or the user's home directory are writable by other users, then the file could be modified or replaced by unauthorized users. In this case, sshd will not allow it to be used unless the StrictModes option has been set to "no"." StrictModes defaults to yes.

Example (illustrative): On web01, deploy logs in with a new Ed25519 key. The client prints deploy@203.0.113.10: Permission denied (publickey). and ssh -v shows the key was offered. The server log shows Authentication refused: bad ownership or modes for directory /home/deploy. ls -ld /home/deploy shows mode 775. In octal, 775 AND 022 is 020, which is not zero, so group write is enough to trigger the refusal. The key was fine; the home directory was group-writable because a shared deployment group had been given write access to it.

Fix

Ownership and modes

Make only the owner able to write. Ubuntu's documentation uses chmod go-w .ssh/authorized_keys, and the sshd manual recommends permissions of "read/write for the user, and not accessible by others" for the file. For a directory named in the log:

chmod go-w /home/deploy
chmod go-w /home/deploy/.ssh
chmod go-w /home/deploy/.ssh/authorized_keys

Risk: removing group write from a home directory can break a shared workflow that depended on it. Check who uses the directory first. Back out by restoring the previous mode, which you noted from ls -ld. If the log names a path owned by someone else, the check accepts only the user or root as owner, so correct the owner with chown (chown [OPTION]... [OWNER][:[GROUP]] FILE..., per chown(1)) after finding out why it differs. Risk: a chown changes who controls the file and may break a service or person that relied on the old owner; note the old owner from ls -ld first and set it back to undo. Do not set StrictModes no to bypass the check: the manual calls the check "normally desirable", and turning it off lets any user who can write the directory replace the key file.

Key not where sshd looks

The default for AuthorizedKeysFile is .ssh/authorized_keys .ssh/authorized_keys2, and the setting can be none, or point elsewhere. Install the key in a file sshd -T lists, with the right owner, then test again.

Risk: installing a public key grants that key's holder access to the account. Add only a key you have verified and back out by removing that line from the file.

Access settings

If the log says not listed in AllowUsers, either add the user to the list or use a group, and reload sshd. AllowUsers "login is allowed only for user names that match one of the patterns", and "AllowUsers is not consulted for users matched by DenyUsers". Changing this file is an access decision.

Disruptive step: editing sshd_config can lock out everyone, including you. Test with sshd -t first, keep an existing session open, and reload rather than restart; see "Changed sshd_config and now SSH says "Connection refused" on port 22: prevent it and recover". Back out by restoring the saved copy of the file.

Key algorithm

If the log shows signature algorithm ... not in PubkeyAcceptedAlgorithms, the server's list does not include the key's signature type. Create and install a key of a type the server accepts (Ubuntu's documentation recommends Ed25519, ssh-keygen -t ed25519). Widening the server list weakens it, and should be a deliberate decision.

Locked account or shell

If the log says the account is locked, or that the shell does not exist or is not executable, fix the account, not the key.

Confirm the fix

  1. The server log shows Accepted publickey for deploy from <ip> port <port> ssh2, followed by key details.
  2. The client login works without a password prompt, and a second login works too.
  3. sshd -T -C user=deploy,host=client.example.com,addr=198.51.100.7 shows the settings you intended.

What to watch afterwards

  • Repeated failures. Treat failures for your own accounts as a fault to chase, and decide for yourself how to treat Failed publickey lines from addresses you do not know.
  • Group-writable homes. A deployment change that adds group write to a home directory will bring the refusal back.
  • Leavers. Remove keys when people leave; see "Revoke a Departed Administrator's Access Across a Hosted Estate".
  • Log level. Debug logging "violates the privacy of users and is not recommended" according to the sshd_config manual; use it briefly if at all.

Risks and limits

  • These commands have not been run or lab-tested. They come from the OpenSSH source and manual pages and Ubuntu documentation listed below.
  • The log messages come from OpenSSH portable at master. Distribution builds can reword them, and SELinux or other access controls can also refuse a key file without these messages. The sources used do not cover those controls.
  • A key rejected by PubkeyAcceptedAlgorithms because of an old signature type is covered only as the log message; the history of which algorithms were removed in which release is not covered.
  • Authentication through certificates, AuthorizedKeysCommand or external directories adds other failure paths that are not described here.

Sources