Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

ESXi root locked: 'Remote access for ESXi local user account 'root' has been locked for 900 seconds'

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

If the vSphere Host Client or SSH rejects the root password you know is correct, and /var/run/log/vobd.log on the host holds the message "Remote access for ESXi local user account 'root' has been locked for 900 seconds after" a number of failed login attempts, the account has been locked by the host's failed-login protection. The sender need not be a person. It can be another system that keeps sending a wrong root password, for example a monitoring server or backup appliance still using the password from before the last rotation.

Waiting 15 minutes unlocks the account, but the lock returns if the sender is still trying. Find the sender first, then unlock.

What the symptom means

Broadcom documents the behaviour in its ESXi knowledge base. Account locking applies to access through SSH and through the vSphere Web Services SDK. By default a maximum of five failed attempts is allowed before the account is locked, and the account is unlocked after 15 minutes. Three advanced options control it:

  • Security.AccountLockFailures: the maximum number of failed login attempts before a user's account is locked. Zero disables account locking.
  • Security.AccountUnlockTime: the number of seconds a user is locked out. The default is 900 seconds, which is 900 ÷ 60 = 15 minutes.
  • Security.PasswordHistory: the number of passwords remembered for each user. This one is not part of lockout.

The lock message is written to vobd.log as both vob.user.account.locked and esx.audit.account.locked, and Broadcom's sample shows the first form twice and the second once. The number of seconds and the count of failed attempts in your message may differ from the 900 and the small number above, because the options are changeable and the counter keeps rising while the sender keeps trying. One Broadcom article shows a host locked for 120 seconds after 44090 failed login attempts. Another shows 52.

Broadcom's pages differ on the console, and three of them agree against one. KB 326765 and KB 312772 say the Direct Console Interface (DCUI) and the ESXi Shell do not support account lockout, and KB 378714 says root can still sign in through the DCUI while SSH and the web interface are locked. Only KB 446675, on a standalone host, says users could not sign in through the DCUI, SSH or the Host Client during a lockout. The fix below follows the DCUI route, so treat the DCUI as the route to try first, not a guaranteed one.

Lockdown mode is a different control that also stops direct access. If your message is 'Permission to perform this operation was denied' rather than the lock message above, read "ESXi Lockdown Mode: 'Permission to perform this operation was denied' and Other Ways You Lose Direct Access to a Host" instead.

Likely causes

Cause What you see How to confirm
A monitoring, backup or inventory tool still uses the old root password Lock appears soon after a root password change; returns after every unlock Repeated "Rejected password for user root from" lines with the same address in hostd.log
An automated scanner or unauthorised login attempts Many attempts from an address you do not recognise Source addresses in auth.log and the host's Events tab
A script or appliance with a stale saved credential, for example a cluster management VM Count of failed attempts in the tens of thousands The same source in hostd.log; the failed count in vobd.log keeps growing
The lock timer was set very high, or the lockout counter is being reset on a schedule Account unlocks at odd times, or never when you expect The two option values, and the six-hourly reset described under What to watch
Lockdown mode or a permissions change, not a lock 'Permission to perform this operation was denied' The lockdown mode article named above, not this one

Prove the cause before changing anything

All of these are read-only. You need a console route that does not depend on the root password working over the network: the DCUI through the server's out-of-band console (iDRAC, iLO, CIMC or a KVM), or an SSH session that is still open.

  1. Confirm the lock. On the host, search /var/run/log/vobd.log for vob.user.account.locked. The message includes the lock time in seconds and the number of failed attempts.
  2. Find the sender in auth.log. Broadcom's article shows the lines to look for in /var/run/log/auth.log: "Connection from" followed by an address and port, a pam_tally2(sshd:auth) line with the user, a tally and a deny count, and "error: PAM: Authentication failure for root from" followed by an address. A tally that keeps rising while a single address repeats is the signature of a sender with the wrong password.
  3. Cross-check in hostd.log. In /var/run/log/hostd.log, look for "Rejected password for user root from" and an address. Broadcom's article for the 120-second case also points to syslog.log and rhttpproxy.log in /var/run/log/ for further detail.
  4. Use the Events tab. Another Broadcom article says you can sign in to the host's web interface after unlocking, go to Monitor then Events, and read the addresses that attempted to log in. That list also includes the attempts you just made yourself.
  5. Read the current counter. From the ESXi Shell, pam_tally2 --user root prints the login failures, the latest failure time and the source (which can be "unknown").
  6. Read the two option values. In the vSphere Client select the host, go to Configure, System, Advanced System Settings, click Edit and filter for Security.AccountUnlockTime or Security.AccountLockFailures.

Example (illustrative): esx01 was given a new root password on Monday. On Tuesday morning vobd.log shows the lock message with a count of 87. In hostd.log every "Rejected password for user root from" line carries 192.0.2.50. That address belongs to mon01, a monitoring server that polls every 30 seconds with the old password. With a lock threshold of five failures, five polls take 5 × 30 = 150 seconds, so the account re-locks 2.5 minutes after each unlock. Changing the stored password on mon01 ends it. Raising the lock time would only have hidden it.

Fix

Step 1: stop the sender

Identify the service or device from the source address and give it the correct credential. Broadcom's advice for a tool that is no longer needed is to disable its authentication requests. Blocking the address on a network firewall is a second option Broadcom mentions. Do this first. Unlocking the account while the sender is still running only restarts the lock.

Step 2: unlock root

Disruptive step: this enables the ESXi Shell on the host and clears the failure counter. The shell stays enabled until you turn it off, which weakens the host's access controls. Back out by disabling the ESXi Shell again under Troubleshooting Options when you finish. Clearing the counter itself cannot be undone, but the counter is only a record of failures.

  1. Reach the DCUI through the out-of-band console and press F2 to sign in as root.
  2. Go to Troubleshooting Options and choose Enable ESXi Shell, then return to the main menu.
  3. Switch to the shell: Broadcom's articles give Alt+F1 and Ctrl+Alt+F1 for this step. Use whichever your console accepts.
  4. Run pam_tally2 --user root to see the count, then pam_tally2 --user root --reset. A repeat of the first command should show zero failures for root.
  5. Sign in again through the Host Client or SSH.

If the sender has already been fixed, the alternative is to wait for Security.AccountUnlockTime to pass. Broadcom's note on a standalone host says to stop the source and wait, and gives the default as 15 minutes, up to 30 minutes depending on configuration.

Step 3 (only if you must): pause the lock while you fix many hosts

For a fleet where every host is locked because a third-party service or tool (for example a cluster management VM) holds the old password, Broadcom's 120-second article gives a temporary workaround: set Security.AccountLockFailures to 0 to disable locking after failed logins, repair the sender, unlock each host, then put the option back to its original value (Broadcom's text gives "typically 5").

Risk: with the option at 0, nothing slows an attacker who is guessing the root password over SSH. Keep the window short, record the original value first, and restore it. To back out, set the option back to the value you recorded and confirm it in Advanced System Settings.

Confirm the fix

  • The lock message stops appearing in vobd.log after the unlock time.
  • The "Rejected password for user root from" lines stop in hostd.log.
  • pam_tally2 --user root shows no failures building up over 10 to 15 minutes.
  • Root signs in over SSH and the Host Client, and the ESXi Shell is switched off again.

What to watch afterwards

  • After every root password rotation, list the systems that log in to hosts as root and update them in the same change. Broadcom's own articles name the same pattern: a root password change followed by a monitoring or backup server still using the old one.
  • Set an alert on the esx.audit.account.locked message. It is the earliest sign that something is sending bad credentials.
  • Broadcom documents that from ESXi 8.0 Update 2 a cron job on the host runs pam_tally2 --reset every six hours. It was introduced as a temporary measure against accidental lockouts. If your policy needs a lock that lasts longer than six hours, the effective maximum is six hours, whatever Security.AccountUnlockTime says. Broadcom publishes a procedure to disable that job, which edits a system file and a startup script; apply it only through your change process.
  • Do not use Security.AccountLockFailures set to 0 as a permanent setting.

Risks and limits

  • These commands have not been run or lab-tested. They come from the Broadcom knowledge base articles listed below.
  • Broadcom's pages differ on whether the DCUI is exempt from lockout (KB 326765, 312772 and 378714 say or show that it is; only KB 446675 reports a failed DCUI sign-in) and on the default wait: KB 446675 says "by default, ESXi locks accounts for 15 minutes after 5 to 10 failed attempts, but this may extend up to 30 minutes depending on configuration", while KB 326765 states a maximum of five failed attempts and 15 minutes.
  • The advanced option names and defaults are taken from Broadcom's knowledge base text. Where a build behaves differently, trust the value shown in Advanced System Settings.
  • Resetting the counter does not tell you who was sending the bad logins. Do not skip the log checks.
  • Enabling the ESXi Shell and SSH is a security trade-off. Turn them off again after the work.

Sources