About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.
vSphere Cluster Services (vCLS) runs small agent VMs in each cluster so DRS keeps working. TechDocs: "DRS depends on the availability of vCLS VMs." When those VMs are missing, the cluster's vSphere Cluster Services health turns Degraded, then Unhealthy, and DRS stops making moves. TechDocs is clear that this is not the same as DRS being switched off: "If DRS is non-functional this does not mean that DRS is deactivated." Before you change anything, find out which kind of vCLS the cluster runs. The logs, the causes and the fixes are different for each.
What the error means
vCenter raises events and alarms like these. KB 395903:
Cluster Agent VM is missing on cluster (vCLS), Event Type ID com.vmware.vim.eam.issue.cluster.agent.vmNotDeployed.
with "The vSphere Cluster Services health status is reported as Degraded." KB 316514 quotes the DRS alarm: "vsphere drs functionality was impacted due to unhealthy state vsphere cluster services caused by the unavailability of vsphere cluster service vms."
The health states are defined in TechDocs:
| State | Meaning |
|---|---|
| Healthy (green) | "If there is at least one vCLS VM running, the status remains healthy, regardless of the number of hosts in the cluster." |
| Degraded (yellow) | "If there is no vCLS VM running for less than 3 minutes (180 seconds), the status is degraded." |
| Unhealthy (red) | "If there is no vCLS VM running for 3 minutes or more, the status is unhealthy in a DRS enabled cluster." |
There are two kinds of vCLS. TechDocs: "vCLS is now referred to External vCLS for releases earlier than vSphere 8.0 Update 3. vCLS starting with vSphere 8.0 Update 3 and later is now referred to as Embedded vCLS." "Embedded vCLS is activated once the vCenter Server is 8.0 Update 3 and in clusters that have one or more 8.0 Update 3 ESXi hosts." External vCLS runs "up to three vCLS VMs"; Embedded runs "up to two". The vCenter 8.0 Update 3 release notes say that with Embedded vCLS, "Deployment and lifecycle of these VMs are managed within ESXi and are no longer managed by the vSphere ESX Agent Manager (EAM)." And "Embedded vCLS does not utilize datastores and ignores the placement settings."
Likely causes
| Cause | vCLS type | Where it shows | Source |
|---|---|---|---|
| EAM stuck deploying External VMs after the cluster moved to Embedded | Mixed / 8.0 U3 | eam.log every 30 seconds | KB 395903 |
| Invalid ESXi hostname (trailing dot, dash, space) | Embedded | infravisor.log, vpxd.log | KB 378133, 425726, 404681 |
| No suitable datastore | External only (a cluster with no 8.0 Update 3 host) | eam.log "lack of suitable datastore" | KB 395903, TechDocs |
| Stale vCLS VMs ("Duplicate key") | External | eam.log | KB 326366 |
| Cluster left in Retreat Mode | Both | Cluster Summary warning | KB 404022, TechDocs |
| Someone powered off, moved or deleted the VMs | External. On Embedded, vSphere restarts a powered-off VM | Events | TechDocs |
Prove the cause before changing anything
These steps only read.
1. Read the health state. TechDocs: "You can monitor the health status of vCLS in the Cluster Services portlet displayed in the Summary tab of the cluster." Note whether it says Degraded or Unhealthy and for how long.
2. Check the vCLS mode. Open Cluster > Configure > vSphere Cluster Services. If the mode shows Retreat Mode, vCLS has been switched off on purpose and KB 404022's alarm applies: "vSphere Cluster Services are disabled by Retreat Mode. vSphere DRS is not functioning and vSphere HA will not perform optimally". To tell which kind of vCLS the cluster runs, use the rule above: if vCenter Server is 8.0 Update 3 and any ESXi host in the cluster is 8.0 Update 3, the cluster is Embedded. Otherwise it is External. TechDocs also gives the cluster property vclsVmType (Embedded or external).
3. Find the VMs. TechDocs: "vCLS VMs are not displayed in the inventory tree in the Hosts and Clusters tab." They sit in a folder named vCLS in the VMs and Templates tab. KB 316514: "vCLS VMs are named vCLS (<number>)".
4. Read eam.log on vCenter. In /var/log/vmware/eam/eam.log, KB 395903 shows repeating lines ending:
Can't provision VM for ClusterAgent(ID: ) due to lack of suitable datastore.
Read this line by cluster type. KB 395903: "If any ESXi host in the cluster is upgraded to 8.0 U3, the cluster vCLS mode changes from External vCLS to Embedded vCLS." On a cluster with any 8.0 Update 3 host, the line is not a datastore fault. It is EAM retrying External deployments on a cluster that has become Embedded, and the fix is the Retreat Mode toggle below. Only on a cluster with no 8.0 Update 3 host does it mean there is no usable datastore.
KB 326366 shows java.lang.IllegalStateException: Duplicate key VirtualMachine:vm-621463 when stale VMs block a new deployment.
5. Embedded vCLS: read the host side. KB 378133 lists /var/run/log/infravisor.log on the ESXi host, where an invalid hostname shows as:
ValidatePodCreate failed: [spec.nodeName: Invalid value: \"esx.\": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character ...
vCenter's vpxd.log shows VclsPodCrxReconfigure(reason: 'VM power-on timeout on host-1##'). Check each host's name with esxcli system hostname get (KB 425726) and cat /etc/hosts (KB 378133). KB 425726's case: "The ESXi host configuration contains a trailing dot at the end of the FQDN (example, esxi01.domain.local.)". KB 404681 prints two more read-only checks:
configstorecli config current get -c esx -g infravisor_pods -k vcls
inf-cli get pods -n vcls
KB 425726 describes the fault as "the vCLS pods remain absent from the cluster", so a host that lists no vCLS pod is the affected one.
6. Check recent operations. TechDocs lists actions that break vCLS, among them "Changing the power state of the vCLS VMs", "Triggering vMotion of the vCLS VMs", "Deleting the vCLS VMs from disk" and "Moving vCLS VMs to a resource pool". Check the cluster's events for any of these.
Example (illustrative): in a three-host cluster, one host is patched to 8.0 Update 3, the health turns Unhealthy, and eam.log repeats the "lack of suitable datastore" line every 30 seconds. That is KB 395903's case: use the Retreat Mode toggle below.
Fix
EAM stuck after the move to Embedded (KB 395903). "Perform a Retreat Mode toggle to force a fresh reconciliation". Go to Cluster > Configure > vSphere Cluster Services > Edit Settings and set the mode to Retreat Mode. "Confirm that the vSphere Cluster Services health shows Degraded and any stale vCLS VM records are removed." Then "change the mode to System Managed" and check that the Embedded VMs deploy. Disruptive step: "vSphere HA and DRS functionalities are impacted while the cluster is in Retreat Mode." The effects are listed under the hostname fix below. Way back: set the mode to System Managed.
Invalid hostname on Embedded vCLS (KB 378133). KB 378133 notes that "Entering and exiting Retreat mode option will not deploy new vCLS VMs" for this cause. Its order: enable Retreat Mode, put the host in maintenance mode, set a valid name, check /etc/hosts, restart the agents, exit maintenance mode, repeat per host, then return to System Managed:
esxcli system hostname set --host esx --domain example.com
services.sh restart
Risk: with Retreat Mode on, DRS does not run. KB 316514 (7.x and 8.x): "When a user wants to take down a host for maintenance, running VMs will not be automatically migrated to other hosts within that cluster." Before you enter maintenance mode, migrate the host's VMs by hand or shut them down. Workloads are also not load-balanced, and "HA will still power-on the VMs but these VMs might be powered on in a less optimal host." Way back: the last step returns the cluster to System Managed.
Disruptive step: the host restart. KB 378133 warns "The host can be shown as "not responding" in vCenter for a short period of time" after services.sh restart.
If the pods are still absent after the hostname fix. KB 425726: "Enabling and disabling Retreat Mode successfully removes and recreates the vCLS ESX Agent, however the vCLS pods remain absent from the cluster." On the same host, KB 404681 and KB 425726 add:
configstorecli config current set -c esx -g infravisor_pods -k vcls -p /pod_settings/enabled -v true
/etc/init.d/hostd restart
/etc/init.d/vpxa restart
/etc/init.d/infravisor restart
inf-cli kill -p /etc/vmware/infravisor/manifests/vcls.yaml
inf-cli get pods -n vcls
Disruptive step: KB 404681 restarts hostd, vpxa and infravisor on the host, so expect it to show as not responding in vCenter for a short time. The two KBs differ: KB 425726 sets the pod setting only "If disabled" and restarts only infravisor. Stop when inf-cli get pods -n vcls lists the pod.
No suitable datastore (External vCLS). TechDocs: "The datastore for vCLS VMs is automatically selected based on ranking all the datastores connected to the hosts inside the cluster." Give the hosts a shared datastore they can all reach. KB 396571 warns: "It is strongly not recommended to migrate vCLS VMs to another datastore via vMotion". When a datastore holding vCLS VMs goes into maintenance mode, TechDocs says "you must manually storage vMotion the vCLS VMs to a new location or put the cluster in retreat mode."
Stale VMs (KB 326366). "There is currently no automated resolution for this issue." The KB's workaround, in order:
- Take the vCenter backup first. Risk: "An offline snapshot or file-based backup of the vCenter Server must be taken prior to performing these steps."
- Copy the cluster's
domain-c<number>from the browser URL. On the vCenter Server object, open Configure > Advanced Settings > Edit Settings and addconfig.vcls.clusters.domain-c<number>.enabledwith the valueFalse. The cluster is now in Retreat Mode, so the Retreat Mode effects above apply. - Power off all vCLS VMs on the cluster, then right-click each and choose Delete from Disk. Disruptive step: Delete from Disk cannot be undone, so select only vCLS VMs of the affected cluster.
- If a VM cannot be deleted in the vSphere Client, KB 326366 refers you to a separate article on removing orphaned VM records from the vCenter database (VCDB). Risk: that is a database edit, and the way back is the backup from step 1.
- Set the same advanced setting to
True. "The vCLS VMs will be recreated automatically within a few minutes." Repeat for any other affected cluster.
Cluster left in Retreat Mode (KB 404022). Use EDIT VCLS MODE and select System Managed. TechDocs: "DRS is not functional, even if it is activated, until vCLS is reconfigured by removing it from Retreat Mode."
Older vCenter without the UI option. KB 316514: "Starting in vSphere 7.0 U3o and 8.0 U2, entering Retreat Mode is now available as a Cluster setting within the vCenter Server UI." Before that, the advanced setting config.vcls.clusters.domain-c<number>.enabled is set to False. Risk: KB 316514 warns that wrong values "will result in vpxd failing to start". Copy the domain ID exactly. Way back if vpxd no longer starts: take a backup of /etc/vmware-vpx/vpxd.cfg, then run sed '/<vcls>/,/<\/vcls>/d' -i /etc/vmware-vpx/vpxd.cfg. KB 316514: "This will remove all Retreat Mode settings from all of the clusters in this vCenter, but it will allow vpxd to start again."
Confirm the fix
- The Cluster Services portlet shows Healthy.
- The vCLS folder holds the expected VMs: up to three External or up to two Embedded.
- No new failures in eam.log or infravisor.log.
- The DRS warning has gone from the Cluster Summary.
What to watch afterwards
- Host upgrades to 8.0 Update 3 in a cluster that still has older hosts. The vCLS mode changes when the first host is upgraded (KB 395903).
- New hosts: check the hostname has no trailing dot before adding them.
- Retreat Mode left on after maintenance.
- Staff actions on the vCLS folder. TechDocs: "Because vCLS VMs are treated as system VMs, you do not need to backup or snapshot these VMs."
Read next
Related articles you may want next.
- VMwareDRS pauses when the only Embedded vCLS host is unavailableUse vCLS transition events and host availability to explain a DRS interruption in a mixed ESXi 8.0 U3 cluster without deleting service VMs.Read article
- VMwareDatastore "Is in Use" and Will Not Unmount: Record the Consumers Before You Change AnythingvSphere 8.0 datastore won't unmount, resource in use? Record inventory objects, host settings and open files per host, with evidence strength, before any change.Read article
- VMwareESXi host stuck entering maintenance mode at 2%, 19% or 29–32%ESXi host stuck entering maintenance mode? Use the percentage to find the cause: precheck at 2%, VM moves at 19%, DRS or vCLS at 29–32%. Prove it first.Read article
Risks and limits
- KB 316514: "On vSphere versions prior to vSphere 9.0, being in Retreat Mode will impact some of the cluster services for that cluster." It also says Retreat Mode "will also delete the vCLS VMs." The DRS and HA effects above apply to vSphere 7.x and 8.x. KB 316514 states that from vSphere 9.0 vCLS "can be disabled on a vSphere cluster without any impact to vSphere DRS or vSphere HA functionality".
- KB 316514: "Using Retreat Mode on a vSAN cluster will show Cluster Service health as "Degraded"."
- The release notes list a known issue: the HA status of Embedded vCLS VMs shows as Unprotected, and "this is expected, because vSphere HA does not manage Embedded vSphere Cluster Service VMs."
- For a cluster partly upgraded to 8.0 Update 3, the related article on the mixed-cluster DRS gap covers the transition window.
NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.
Sources
- vSphere Cluster Services (vCLS) — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- Embedded vCLS — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- Monitoring vSphere Cluster Services — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- Maintaining Health of vSphere Cluster Services — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- vSphere DRS and vCLS VMs — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- Putting a Cluster in Retreat Mode — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- Datastore selection for External vCLS — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- VMware vCenter Server 8.0 Update 3 Release Notes — Broadcom TechDocs, Last Updated October 28, 2026, accessed 17 November 2026
- Disable vCLS on a Cluster via Retreat Mode (KB 316514) — Broadcom, Updated On 07-21-2026, accessed 17 November 2026
- Error: Cluster Agent VM is missing on cluster (vCLS) in VMware vSphere 8.0 (KB 395903) — Broadcom, Updated On 10-20-2026, accessed 17 November 2026
- Embedded vCLS machines will not deploy in vSphere 8.0 Update 3 (KB 378133) — Broadcom, Updated On 08-12-2026, accessed 17 November 2026
- DRS not functional/vCLS VM's not deploying post patching ESXI hosts to 8.0 update 3 (KB 404681) — Broadcom, Updated On 05-26-2026, accessed 17 November 2026
- vCLS pods fail to deploy in a DRS and HA enabled cluster (KB 425726) — Broadcom, Updated On 02-13-2026, accessed 17 November 2026
- Deployment of vCLS VMs Fails with "Duplicate key" Exception Due to Stale VMs (KB 326366) — Broadcom, Updated On 06-25-2026, accessed 17 November 2026
- vSphere Cluster Services are disabled by Retreat Mode. vSphere DRS is not functioning and vSphere HA will not perform optimally (KB 404022) — Broadcom, Updated On 07-13-2025, accessed 17 November 2026
- Migrating vCLS VM to another datastore (KB 396571) — Broadcom, Updated On 12-15-2025, accessed 17 November 2026
This article was originally published on .