Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

ESXi Lockdown Mode: 'Permission to perform this operation was denied' and Other Ways You Lose Direct Access to a Host

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

If root, or any other account, is refused at the ESXi Host Client, over SSH or at the console menu, and the message reads "Permission to perform this operation was denied.", the host is almost certainly in lockdown mode. Lockdown is working as designed. Virtual machines keep running; only direct management is blocked, as Broadcom KB 371627 describes.

The way back depends on two facts: whether the host is in normal or strict lockdown, and whether vCenter Server can still reach it. If vCenter can manage the host, you can switch lockdown off from the vSphere Client in a few clicks. If vCenter is gone, normal lockdown can still be switched off at the console, but strict lockdown may leave no way in at all.

What the error means

Lockdown mode makes the host take its management instructions from vCenter Server. Broadcom's 8.0 lockdown documentation says there are two levels, normal and strict, and that "Exception users do not lose their privileges when the host enters lockdown mode".

The two levels differ in one way that matters when something breaks. According to the lockdown mode behaviour page (8.0), in strict lockdown mode the Direct Console User Interface (DCUI) service is deactivated. In normal lockdown it keeps running, and accounts on the Exception User list that have administrator privileges, plus users listed in the DCUI.Access advanced setting, can still use it. The same page says that sessions already open in the ESXi Shell or over SSH survive the switch to lockdown only for exception users with administrator privileges.

KB 336894 adds that the "Configure Lockdown Mode" option in the DCUI is greyed out if vCenter is down or the host is disconnected, and gives the warning that matters most in recovery: if lockdown is enabled or disabled from the DCUI, the permissions for users and groups on the host are discarded.

Who may log in during lockdown is narrow. The Broadcom 8.0 behaviour page lists vCenter's own account (vpxuser), exception users with permissions, and the vCloud Director account (vslauser, if present) for the vSphere web services API. Ordinary local accounts, root included, are not on that list.

Likely causes

Cause What you see How to confirm
Host is in lockdown and you are using a normal account "Permission to perform this operation was denied." at the Host Client or SSH The host's Lockdown Mode setting in the vSphere Client reads Normal or Strict
Exception user exists but has no permission on the host permission is denied for the exception user at web UI, SSH or DCUI esxcli system account list shows the account, but the host has no permission assigned for it (KB 395362)
Exception user is an Active Directory group or a vCenter user Login fails even though the name is on the list The 7.0 exception user page says exception users must be host-local users, or AD users with privileges defined on the host, not AD group members or vCenter users
vCenter lost, host in strict lockdown DCUI is not available; no accepted login DCUI shows no Configure Lockdown Mode option or will not start; vCenter shows the host disconnected
vCenter lost, host in normal lockdown Host Client and SSH refuse root; DCUI works for authorised users F2 at the console works for root or a user in DCUI.Access
Lockdown cannot be turned on at all "Internal Error Occurred" in vCenter, or "A general system error occurred: Internal error" from the CLI An orphaned domain group exists on the host (KB 388685)

Prove the cause before changing anything

Everything here is read-only. These commands have not been run or lab-tested.

  1. Read the lockdown setting. In the vSphere Client, select the host, then Configure > System > Security Profile, and read the Lockdown Mode panel. KB 371627 uses the same path. The panel shows Disabled, Normal or Strict.
  2. Check the host's connection state in vCenter. A host shown as not responding or disconnected is the starting point for the recovery routes below. If vCenter cannot reach the host at all, fix that connection first or use the console routes below.
  3. Check the console. At the DCUI, press F2 and sign in. If Configure Lockdown Mode appears, you are in normal lockdown (or lockdown is off). If the DCUI does not respond for any account, suspect strict lockdown.
  4. Check the exception user. If the exception user is being refused, you need a shell on the host. Strict lockdown allows exception users with administrator privileges to use SSH if the SSH service has been turned on. From such a session, or from the console in normal lockdown, KB 395362 gives the first command:
esxcli system account list

It lists the host's local accounts. The exception user must be on that list. Broadcom does not publish the output layout in that article, so read it for the account's ID. A user who is not listed does not exist on the host. A domain user will not appear here; check the permission under the host in vCenter (Inventory > Cluster > Host > Permissions) instead.

  1. Check the exception user's type. The 7.0 exception user page states that exception users must be local host users or AD users with privileges defined locally, not AD group members or vCenter users. That page is the 7.0 text; check the wording for your own release.

Example (illustrative): esx01 is a host in cluster prod-a, managed by vc01. Backups through a service account svc-backup began failing the day lockdown was enabled. In the vSphere Client the Lockdown Mode panel reads Normal, and svc-backup is on the Exception Users list. At the DCUI, svc-backup is refused, while root can use F2. Running esxcli system account list over SSH, started by root before lockdown was set, shows svc-backup as a local account, so it exists. The missing piece is a permission on the host itself.

Fix

Cause 1: host is in lockdown and vCenter is healthy

This is the cleanest route. In the vSphere Client, go to the host's Configure > System > Security Profile, edit the Lockdown Mode panel and set it to Disabled (KB 371627, option 1). Doing it through vCenter keeps the host's user and group permissions, which the DCUI route does not (see below).

If you only need an administrator to reach the host for a short time, consider adding a properly scoped exception user instead of switching lockdown off, since the 7.0 page says the exception list is meant for service accounts that perform very specific tasks, not for administrators, and that adding administrators defeats the purpose of lockdown.

KB 336894 documents a PowerCLI route for scripted work: (get-vmhost <hostname> | get-view).EnterLockdownMode() and .ExitLockdownMode(). The same article says its older vim-cmd forms apply to ESXi 5.x and 4.1 only, so do not use them on current releases.

Risk: switching lockdown off widens the set of accounts that can log in directly to the host. Turn it back on as soon as the work is done, and record who made the change.

Cause 2: exception user is refused

Per KB 395362, the cause is that the exception user or group has no permission on the host. The steps there are:

  1. If lockdown is on, turn it off first, from vCenter.
  2. For a local user, create the account under the Host Client's Manage > Security & users > Users and set a password. For a domain user, assign permissions in vCenter under Inventory > Cluster > Host > Permissions.
  3. For a local user, over SSH:
esxcli system account list
esxcli system permission set --id=<Exception userid/group> --role=Admin
  1. In vCenter, open the host's Configure > Security profile > Edit, enable Lockdown mode (Normal or Strict), add the exception users and save.

Broadcom notes that in normal lockdown the exception user can then reach the web UI, SSH and the DCUI, and in strict lockdown the web UI and SSH only, because the DCUI service is stopped.

Risk: the Admin role gives full control of the host. The exception list is for narrow service accounts. If the account only needs to read, assign the narrowest role your process allows; KB 395362 shows Admin and does not discuss lesser roles.

Cause 3: vCenter is lost and the host is in normal lockdown

KB 371627, option 2: reach the console through the out-of-band controller (iDRAC, iLO, KVM) or a local monitor, press F2, authenticate with root or an authorised DCUI user, choose Configure Lockdown Mode, toggle it to Disabled, then save and exit.

Risk: per KB 336894, changing lockdown from the DCUI discards user and group permissions on the host. After this step, reapply any exception user permissions (Cause 2) before you re-enable lockdown from vCenter.

Cause 4: vCenter is lost and the host is in strict lockdown

Broadcom's KB 371627 is blunt about this one. With the DCUI disabled, and no SSH or ESXi Shell access prepared for exception users, the options are to restore connectivity to vCenter Server or to reinstall the host. If SSH or the shell was left on and an exception user with permissions exists, use that account: log in, then work through Cause 2. Reinstalling a host is a destructive, last-resort step. Check with whoever owns the datastores before taking it.

KB 336894 gives the same warning and adds that troubleshooting services (SSH, the shell, DCUI) are off by default and can be turned on whether or not lockdown is active.

Cause 5: lockdown will not turn on

KB 388685 covers a host with an orphaned domain group. The sequence it gives, for ESXi 7.x and 8.x, is: SSH to the host as root; try to leave the domain in the Host Client (Manage > Security & Users > Authentication) and copy the full error text, which names the group; create a local group with exactly that name using /usr/lib/vmware/busybox/bin/busybox addgroup 'DOMAIN\\group^name'; leave the domain again; remove the local group with /usr/lib/vmware/busybox/bin/busybox delgroup 'DOMAIN\\group^name'; rejoin the domain if needed; then retry lockdown.

Disruptive step: leaving and rejoining a domain changes how domain accounts authenticate to the host while it is out of the domain. Do this in a maintenance window and have a local fallback account available.

Confirm the fix

  • In the vSphere Client, the host's Lockdown Mode panel shows the mode you intended.
  • An exception user can sign in to the web UI or SSH (and, in normal lockdown, the DCUI), and root is refused when lockdown is on.
  • vCenter shows the host as connected, and virtual machines are unaffected.
  • If you used the DCUI to switch lockdown off, check the host's permissions list in the vSphere Client and confirm that the users and groups you rely on are still present.

What to watch afterwards

  • Record, for each host, whether it is in normal or strict lockdown and which exception users exist, in a place that survives a vCenter loss.
  • Before strict lockdown, confirm that the exception user can sign in over SSH with the service turned on, and that the permission exists on the host.
  • After any DCUI change to lockdown, check permissions straight away.
  • Treat the exception list as a short list of service accounts.

Risks and limits

  • Broadcom documents esxcli system permission set and esxcli system account list in the KB, but the KB does not show their output, so check the output on your own host before scripting against it.
  • No vendor-documented ESXi shell command to read the lockdown state was found. The vSphere Client panel is the documented check.
  • The service-access table on the 8.0 behaviour page does not split ESXi Shell and SSH cleanly between normal and strict lockdown. KB 395362 states that strict lockdown leaves the web UI and SSH available to exception users.
  • The exception user page cited is for vSphere 7.0. The KBs cover 6.x to 8.x with different statements of scope; test on your version.
  • Reinstalling an ESXi host to recover from strict lockdown loses the host's local configuration.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources