Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

vCenter appliance root password expired: get back in and stop it expiring again

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

The root password of the vCenter Server Appliance (VCSA) expires every 90 days by default, and Broadcom notes that nothing warns you until someone signs in to the appliance as root. If root now fails with the right password, or SSH asks you to change the password before it lets you in, the password has expired or the account has been locked by failed logins. You can fix it without a reboot. If you still know the current password, change it in the appliance management interface (VAMI) with an SSO administrator. If the password is lost, use a shell started by an SSO administrator who is in the right group.

Then change the expiry setting so the same date does not catch you again.

What the symptom means

Broadcom's articles describe two separate things that look alike. Root can be expired, meaning the 90 days have passed and the account will not authenticate "even with the correct password". Or root can be locked, meaning too many failed attempts have tripped the lockout counter. A reset can need both: a new password, and a cleared counter.

Two details matter in practice.

  • The prompt on SSH. Broadcom says that if you still know the current root password, an SSH session prompts you to change it, and the change must be finished within 30 seconds before the authentication attempt times out. Broadcom's article 426611 describes the same prompt as appearing "even if you believe the password should still be valid", because the expiry policy set in the VAMI is what decides.
  • Which command clears the counter. From vCenter 8.0 Update 2 the tool is faillock. For 8.0 Update 1 and earlier it is pam_tally2. Broadcom says pam_tally2 is deprecated in Photon 4.

The appliance has a second local account, the SSO shell user. A SystemConfiguration.BashShellAdministrator member can sign in to the appliance and use sudo without a password. Broadcom documents this as the procedure for a lost or expired root password.

Likely causes

Cause What you see How to confirm
Root password past its 90 days SSH asks you to change the password, or login fails with the correct one chage -l root shows a "Password expires" date in the past (needs a working shell)
Root locked by failed logins Correct, unexpired password still rejected faillock --user root or pam_tally2 --user=root shows failures
Password was reset but expires again within minutes "password changed for root" in the log, then the account is expired again /var/log/audit/sshinfo.log shows "account sso-user has expired (failed to change password)"
Root is not the problem; the certificate is Web UI unavailable with certificate errors Follow "vCenter Sign-In Fails After a Certificate Expires: Find the Store Responsible"

Prove the cause before changing anything

You need one working route into the appliance. If the root password is lost completely, go straight to Route 2. Broadcom's VAMI steps (knowledge base 426611) ask for the Current Password as well as the New Password, so the VAMI route needs the current root password. Route 2 needs an SSO administrator in the right group, not root.

  1. Try the VAMI at https://<vCenter FQDN>:5480 as root. If it rejects root, go on.
  2. If you still know the current root password, sign in to the VAMI as an SSO administrator account. Broadcom's note says the Actions menu offers Change Root Password.
  3. If you can get a shell, check the account state. From the SSO user's bash shell, use sudo chage -l root, because the SSO shell user runs commands as root with sudo. The sample output on Broadcom's page 322247 lists "Last password change", "Password expires", "Password inactive", "Account expires", the minimum and maximum days between changes, and the days of warning; its example shows a maximum of 90 and a warning of 7.
  4. Check the lock counter with sudo faillock --user root on 8.0 Update 2 and later, or sudo pam_tally2 --user=root on earlier releases. Broadcom's page 343642 prints the faillock --user root form; the reset form sudo faillock --user root --reset is on page 321369.
  5. Check /var/log/audit/sshinfo.log if the password keeps expiring. That is the log Broadcom names for re-lockouts.

If your symptom is a broken web interface with certificate errors rather than a root login failure, read "vCenter Sign-In Fails After a Certificate Expires: Find the Store Responsible" instead.

Example (illustrative): vcsa01 had its root password set on 2 September. 2 September plus 90 days is 1 December: 28 days remain in September, then 31 in October and 30 in November, which is 89, so day 90 is 1 December. With the default warning of 7 days the warning period begins on 24 November. Nobody signs in to the appliance shell in that window, so nothing is seen until a patch window on 3 December, when SSH prompts for a new password and times out. The admin who knows the current password uses the VAMI route; if it is lost, the SSO shell route gets back in. Neither needs a reboot.

Fix

Take a snapshot or file-based backup first. Broadcom's page 322247 asks for a backup and a snapshot before its GRUB steps, and page 343642 asks for the same before editing files. The SSO shell route on page 321369 does not ask for one, but a snapshot costs little before any password change. If vCenter is in Enhanced Linked Mode, Broadcom asks for offline (powered-off) snapshots of all replicating nodes together. In environments with SDDC Manager, Broadcom says the password will need to be remediated there if it is changed from what SDDC Manager has stored.

Route 1: reset in the VAMI (no reboot)

Sign in to the VAMI as an SSO administrator, open Actions, choose Change Root Password, and enter the current and new password. If the current password is not known, use Route 2.

Route 2: SSO shell, no reboot

Risk: this changes the root password. Anything that stores the old one (scripts, a password vault, SDDC Manager) will fail to authenticate until updated. Back out by setting the password again the same way. Broadcom also gives a way to restore a previous password, which clears /etc/security/opasswd first; use it only if you need the original value.

  1. SSH to the appliance as an SSO administrator in the SystemConfiguration.BashShellAdministrator group.
  2. If it is the first time, enable the shell: shell.set --enable true, then shell.
  3. Clear the lock: sudo faillock --user root --reset on 8.0 Update 2 and later, or pam_tally2 --user=root --reset on 8.0 Update 1 and earlier.
  4. Set the password: sudo passwd root.
  5. Sign in as root with the new password to check.

If the account is not locked and you are in the appliance shell as a super administrator, Broadcom also lists localaccounts.user.password.update --username root --password, which prompts for the new password twice.

Risk: localaccounts.user.password.update is another root password change, with the same effect on stored copies of the old password. Back out by running it again with the previous value.

Route 3: GRUB, only if nothing else works

Broadcom documents resetting through the boot loader: reboot the appliance, press e at the GRUB menu, add rw init=/bin/bash to the end of the line that starts with Linux, continue with F10, run mount -o remount,rw /, clear the lock, run passwd, run umount / and finish with reboot -f.

Disruptive step: this restarts vCenter and takes it offline while the appliance reboots. Take a backup and snapshot first. There is no way to undo the new password other than setting it again.

Stop it expiring again

Open the VAMI as root, go to Administration, find Password expiration settings and choose Edit. "No" means the password never expires. "Yes" asks for the number of days and an email address for warnings. Broadcom notes that the VAMI insists on a valid email format even when you do not want notifications. Its recommended workaround is a placeholder address such as no-reply@localhost.

From the shell, Broadcom lists chage -I -1 -m 0 -M 99999 -E -1 root to set the root password to never expire. chage -M <days> <username> sets a specific period.

Risk: the chage -I -1 -m 0 -M 99999 -E -1 root command switches off more than password expiry: -I -1 removes the inactivity lock and -E -1 removes any account expiry date, and the VAMI "No" choice also stops all expiry. Back out with chage -M <days> root, or choose "Yes" with a number of days in the VAMI. Whether a never-expiring root password meets your own policy is your decision. A fixed period with the warning email pointed at a mailbox someone reads is the alternative.

If the password expires again almost immediately

Broadcom ties this to appliance files that do not match the expected values. The /etc/passwd entry for root should read root:x:0:0:root:/root:/bin/bash and /etc/default/useradd should contain SHELL=/bin/appliancesh. The read-only checks are cat /etc/passwd |& grep root and cat /etc/default/useradd |& grep SHELL. Broadcom's fix edits those files and reboots the appliance; do not edit them without a snapshot, and treat this as a case to raise with Broadcom support.

Confirm the fix

  • chage -l root shows a "Password expires" date, or "never" if you chose that.
  • Root signs in over SSH and the VAMI without a password prompt.
  • The failure counter is zero.
  • Any system that logs in as root with a stored password has been updated.

What to watch afterwards

  • Put the expiry date in your calendar, or choose a policy that ends the problem. Broadcom says no notification is sent until the root account is used to sign in.
  • Check the SSO shell user too. Broadcom's page 322247 shows chage -l sso-user for the default SSO account, with the same 90-day maximum in its example.
  • Keep one tested way in that does not depend on root, such as the SSO shell account route (Route 2).

Risks and limits

  • These commands have not been run or lab-tested. They come from the Broadcom knowledge base articles listed below.
  • Broadcom's pages do not quote the text of the expired-password prompt, so only the log lines in Broadcom's articles are quoted.
  • The articles write the shell command slightly differently (--enable in one, --enabled in another). Use the form your appliance's help accepts.
  • The GRUB route reboots vCenter. Plan it as an outage.
  • Policy choices such as never-expire passwords are yours; Broadcom documents both.

Sources