Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

ESXi host stuck entering maintenance mode at 2%, 19% or 29–32%

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When an ESXi host sits at "in progress" entering maintenance mode, the percentage on the task tells you which stage is stuck. Broadcom's stage map for vSAN hosts puts the precheck at 2%, moving the VMs off the host at 19%, and vSAN data movement from 20% on. Other Broadcom pages report stalls at 2% and 19% on clusters without vSAN too. A stall between 29% and 32% is documented separately: DRS not set to Fully Automated, or a stale vSphere Cluster Services (vCLS) VM. Read the percentage, prove the one blocker with the checks below, and clear it. Do not restart services first; most stalls are a VM or a setting, not a broken host.

What the error means

Entering maintenance mode needs every running VM off the host. vCenter does that by migrating them with DRS, or it waits for you to move or power them off. The host task waits with a message such as "Waiting for all virtual machines to be powered off or suspended or migrated. In a DRS cluster check the Faults page on the DRS tab for troubleshooting" (KB 318773). KB 318773's cause line: "A host fails to enter maintenance mode because resident virtual machines are unable to power off or migrate successfully."

The stage map is in KB 326563 (Maintenance Mode % progress stages in vSAN nodes). It is written for vSAN nodes (vSAN 7.0.x, 8.0.x, 9.0.x). Stalls at 2% and 19% are also reported for clusters without vSAN: KB 394680 says "typically at 2% or 19%", KB 399791 covers 19% with vCLS, and KB 395406 covers 2%. The stages in the vSAN map:

  • 2% – "Precheck. Task can hang at this % if there are HA/DRS settings that might prevent the host from entering Maintenance Mode."
  • 19% – "vMotion of VMs off the host. If the task is stuck at this % then user may need to intervene and manually migrate or power off (whichever is applicable) the remaining Virtual Machines."
  • 20% – 100% – "Resync and/or migration of data onto other nodes" (vSAN only).

The 29–32% stall comes from two other pages: KB 451053, where "the task stalls between 29% and 32%, and fails to evacuate the host", and KB 441659, where the host "becomes stuck at 32%".

Likely causes

Stuck at Cause What you see How to confirm
2% HA admission control on a two-node cluster DRS Faults: "vSphere HA admission control is preventing the operation." Cluster > Configure > vSphere Availability; host count (KB 394680)
2% Expired vCenter solution user certificates Task stays at 2% vecs-cli "Not After" dates (KB 395406)
2% Supervisor (wcp) service not running "Failed to enter namespaces maintenance mode due to Error: system_error …" service-control --status --all (KB 415949)
19% One VM cannot move: ISO, passthrough, local disk, missing network, rule, capacity Some VMs left on the host Manual migrate of each one; read its compatibility check (KB 326563, 318773)
19% (vSAN, 8.0) VM with a missing .vswp file vMotion fails; missing swap file error for one VM hostd.log: "Unable to the read the descriptor file '/vmfs/volumes/vsan:.../vm-name.vswp' (No such file or directory)" (KB 452706)
19% VMware Tools installer ISO still mounted "Tools installation in progress" VM summary shows the Tools installer connected (KB 310528)
19% vCLS cannot release the host "Waiting for <VM_Name> to complete task: Drm.ExecuteVMotionLRO" for a VM that already moved vpxd.log lines (KB 399791)
19% or later VM–host affinity rule leaves no destination vim.fault.VmHostAffinityRuleViolation in vpxd.log Cluster VM/Host rules (KB 412107)
29–32% DRS Manual, Partially Automated or off VMs stay put; nothing moves Cluster > Configure > vSphere DRS (KB 451053)
32% Orphaned or corrupted vCLS VM vCLS VM in Unknown state Cluster > Monitor > vSphere Cluster Services (KB 441659)
vSAN data stage Not enough hosts for Full Data Migration "General vSAN error. vSAN resource check has failed on host." Evacuation mode chosen; policy needs (KB 326857)

Prove the cause before changing anything

1. Read the percentage and the message. Open the task in Recent Tasks and note both. Then check the cluster's DRS Faults page, as the waiting message says.

2. List what is still on the host. KB 318773: "Check the virtual machines tab on the host to determine if only a partial migration occurred." Those VMs are the blockers. For each one, start a manual migrate to another host and read the compatibility check. KB 326563 lists the common 19% blockers: a passthrough device (for example a GPU), a CD/DVD device pointing to an ISO that no longer exists, a VM network missing on the destination, a disk on a local-only datastore, affinity or anti-affinity rules, and insufficient compute on the destination hosts (overutilisation or HA reservations).

3. Check DRS automation. Cluster > Configure > Services > vSphere DRS. KB 451053: "The Distributed Resource Scheduler (DRS) automation level is set to Manual or Partially Automated." "If no manual migration occurs, the task remains stalled at the 29% mark." "This issue will also occur if DRS is disabled/off."

4. At 2%, check admission control, certificates and services.

  • Two-node cluster with DRS Fully Automated: KB 394680 (2 node ESXi cluster fails to automatically migrate VMs) says the task "hangs (typically at 2% or 19%)" because "Admission Control requires at least three hosts in a cluster to maintain failover redundancy (N+1)." vpxd.log shows "MinPoweredOnCapacity check failed".
  • Certificates: KB 395406 (task gets stuck at 2%) names "Solution User certificates are expired on vCenter Server." Its read-only check on the vCenter appliance:
for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

5. At 19% or 32%, check vCLS. Cluster > Monitor > vSphere Cluster Services (vCLS). KB 441659 (stuck in maintenance mode at 32%): "A corrupted or orphaned vCLS virtual machine is preventing vSphere Cluster Services from functioning normally. Because vCLS is required for vSphere Distributed Resource Scheduler (DRS) operation, DRS cannot automatically evacuate the virtual machines". KB 399791 (stuck at 19% … vCLS control plane dependencies) shows "Waiting for <VM_Name> to complete task: Drm.ExecuteVMotionLRO" for a VM that had already moved, and vpxd.log lines about vCLS trying to disable its components on the host.

6. On vSAN, check the evacuation mode. KB 326857 (vSAN resource check has failed on host) shows that Full Data Migration needs spare hosts: "RAID 5: Requires the addition of a 5th host. RAID 1: Requires the addition of a 4th host. RAID 6: Requires the addition of a 7th host." On the host, KB 326563 gives a read-only progress check during the data stage:

cmmds-tool find -f json -t NODE_DECOM_STATE -u $(cmmds-tool whoami) | grep content

"decomState": 4 means the host is decommissioning and data is moving.

Example (illustrative): esx03 sits at 19% for an hour. Its VMs tab shows one VM left, build-agent. A manual migrate reports the CD/DVD drive points to an ISO that no longer exists. Disconnecting the drive lets DRS move it and the task finishes.

Fix

DRS not fully automated (29–32%). KB 451053: on the cluster, Configure > Services > vSphere DRS > Edit, "Change the Automation Level to Fully Automated." Or move each VM yourself: "Right-click each VM and select Migrate." KB 318773 says to cancel the pending task, change the setting, and try again.

A VM that cannot move (19%). Clear its blocker: disconnect a stale ISO, move a disk off local storage, fix the missing port group on the destination, or power off a VM with a passthrough device. For a mounted Tools installer, KB 310528 (Tools installation in progress): "Right-click the virtual machine and click Guest OS > Unmount VMware Tools Installer", or on the host vim-cmd vmsvc/getallvms then vim-cmd vmsvc/tools.cancelinstall vm_ID. KB 310528 adds: "On Linux guest operating system these steps may not resolve the issue. The virtual machine must be powered off to resolve the issue." Disruptive step: powering off a VM stops its service.

Missing .vswp file (19%, vSAN). KB 452706: "Simply rebooting the Guest OS will not resolve the issue". Shut the guest down gracefully, wait for it to power off, then power it on; the host then creates a new swap descriptor. Test a vMotion, then retry maintenance mode. Disruptive step: the VM is down until it is powered on again.

Affinity rule. KB 412107 (Enter Maintenance Mode task stuck): "Virtual machine-host affinity rules prevent DRS from finding a compatible destination host." Edit the rule so another host qualifies, or move the VM by hand. Risk: the rule may exist for licensing or placement reasons; check with its owner.

Admission control on two nodes (2%). KB 394680: Configure > vSphere Availability > Edit > Admission Control, "Set Define host failover capacity by to Disabled". On vSAN, "select Ensure Accessibility when prompted for the data evacuation mode". Risk: "Disabling Admission Control removes the guarantee that VMs can restart in the event of a host failure during the maintenance window. Re-enable the setting once maintenance is complete."

Expired certificates (2%). KB 395406: "Replace the Solution User certificates using the vCert Tool (using options 3. Manage certificates -> 2. Solution User certificates)", then "Restart all vCenter Services after the certificate replacement": service-control --stop --all && service-control --start --all. Disruptive step: all vCenter services stop and start, so the vSphere Client and API are unavailable until they return. Take a vCenter backup or snapshot first.

Stopped wcp service (2%). KB 415949 matches the message "Failed to enter namespaces maintenance mode due to Error: system_error" with wcp stopped and, in /var/log/vmware/wcp/stdstream.log.stderr, the line "panic: Invalid config file /etc/vmware/wcp/wcpsvc.yaml". Its fix: check /etc/vmware/wcp/wcpsvc.yaml at the line the log names, follow the repair article it links (KB 326225), then service-control --start wcp.

vCLS (19% or 32%). KB 399791 Workaround 1: "Place Cluster vCLS into Retreat Mode", power off the vCLS VM if it is not cleaned up, let the host finish, reboot it, exit maintenance mode, then "Place Cluster vCLS back into System Managed Mode." KB 441659 restarts vpxd on the appliance: service-control --stop vmware-vpxd && service-control --start vmware-vpxd, then checks vCLS Health. Disruptive step: Retreat Mode stops DRS for the cluster while it is on, a vpxd restart drops vSphere Client sessions, and the host reboot takes the host down; the host is already in maintenance mode, so its VMs must be off or moved first. Way back: exit maintenance mode, then set vCLS back to System Managed.

vSAN evacuation mode. KB 326857's workaround: "Place the affected host into Maintenance Mode using the Ensure Accessibility option." Risk: the cluster will "operate in a reduced redundancy state until the host exits Maintenance Mode and the data resyncs back to compliance". KB 435985 (vSAN Data Resync After Exiting Ensure Accessibility) calls No Data Migration "highly discouraged in production environments".

A stale task. KB 318773 restarts the ESXi management agents or, for a stale task, the vsphere-ui and vpxd services, after a snapshot of the vCenter VM. On vSAN hosts, KB 337174 (Exiting an ESXi Host Stuck in Maintenance Mode) warns: "Do not use the services.sh restart command or Restart Management agents from the Direct Console User Interface (DCUI) for vSAN hosts!"

Confirm the fix

  • The host task reaches 100% and the host shows "In Maintenance Mode".
  • On the host, vim-cmd /hostsvc/hostsummary | grep inMaintenanceMode (KB 337174) shows inMaintenanceMode = true.
  • The host's VMs tab is empty, apart from VMs you chose to power off.
  • On vSAN, object health is green once data has moved.
  • vCLS health is healthy, and any setting you changed (DRS level, admission control, Retreat Mode) is back to its normal value.

What to watch afterwards

  • ISOs and devices left connected on VMs after builds and Tools installs.
  • DRS level after cluster changes.
  • Rules and passthrough VMs that pin workloads to one host.
  • Certificate expiry on the vCenter appliance, well ahead of the date.
  • vSAN resync after a host returns, before the next host goes in.

Risks and limits

  • Broadcom pages conflict on restarting agents on vSAN hosts: KB 452706 uses services.sh restart for a vSAN host, while KB 337174 says not to. Follow KB 337174's warning.
  • KB 443213 handles another 2% wcp failure, caused by a machine account password mismatch, with an attached script; take a vCenter snapshot first and involve Broadcom support.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources