Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

'Cannot complete the configuration of the vSphere HA agent on the host': read the second half of the message

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When vSphere HA will not enable on a cluster, the task shows "Cannot complete the configuration of the vSphere HA agent on the host." followed by a second sentence. That second sentence is what separates the causes. "Applying HA VIBs on the cluster encountered a failure" and "Setting desired image spec for cluster failed" each have several documented causes, and the log on the host tells you which one you have.

This is about the agent failing to install or configure. How HA behaves once it is running, such as restarts and admission control, has its own guides: "vSphere HA did not restart some VMs: reading the event trail and checking admission control and isolation settings" and "Maintenance Mode Stalls: "vSphere HA admission control is preventing the operation"".

What the error means

vCenter pushes the HA agent, the Fault Domain Manager (FDM), to each host as a VIB, a software package. In a cluster managed by an image, the push goes through vSphere Lifecycle Manager. The failure can be in any step: the image vCenter holds, the version relationship between vCenter and the host, the network path Lifecycle Manager uses to deliver the VIB, the host's storage, or the agent's own configuration file.

Broadcom's articles show these message forms. Exact text, with host-#### and similar placeholders as Broadcom prints them:

  • "Cannot complete the configuration of the vSphere HA agent on the host. 'Setting desired image spec for cluster failed'"
  • "Cannot complete the configuration of the vSphere HA agent on the host. Applying HA VIBs on the cluster encountered a failure"
  • "Failed installing HA component on the host: host-####."
  • "The operation cannot continue due to downgrade of the following Components: vsphere-fdm."
  • "An internal error occurred while staging/remediating the host"
  • "A general system error occurred: Installing HA components failed on the cluster: domain-c####"

Likely causes

Cause What you see How to confirm
FDM wrongly listed as a component in the cluster image (8.x) HA remediation stuck at 10%; the Update Manager service keeps stopping; "No healthy upstream" on Host updates or Lifecycle Manager The exported cluster image JSON lists vsphere-fdm under components
Stale entries in the vCenter database after an upgrade "Setting desired image spec" error; Image Validation Failed vmware-vum-server.log: "Failed to get cached component. No record"; imageservice.log: "Component vsphere-fdm cannot be found in depot."
vCenter older than its ESXi hosts "The operation cannot continue due to downgrade of the following Components: vsphere-fdm." Host /var/run/log/lifecycle.log: ComponentDowngradeError
Datastore or LUN not ready on a host "An internal error occurred while staging/remediating the host"; df -h errors df -h prints "Cannot open volume"; fdm.log says a datastore is inaccessible
Update Manager firewall rule disabled on hosts HA Agent Unreachable; Lifecycle Manager compliance check hangs at 30% esxcli network firewall ruleset list shows updateManager false
Retired VMware repository URLs in Lifecycle Manager Same "Applying HA VIBs" error Cluster uses legacy URLs such as hostupdate.vmware.com
Duplicate lines in the agent's fdm.cfg Agent will not start; "Failed to start fdm service on host" syslog.log on the host: "Duplicate child: unknownStateMonitorPeriod" (Broadcom's page shows it in syslog.log; its fdm.log excerpt shows a different, hostd fault)

Prove the cause before changing anything

Read-only checks, cheapest first.

  1. The exact message. Open the failed task in the vSphere Client and copy the full text, including the suffix and any host- or domain-c identifier.
  2. Versions. On the host: esxcli software vib list | grep -i fdm. Broadcom says the build number of the vCenter and the vsphere-fdm VIB should match. Compare the vCenter build with the highest ESXi build in the cluster.
  3. Firewall. On the host: esxcli network firewall ruleset list. Check the updateManager line.
  4. Storage. On the host: df -h. Every presented datastore should list. An error such as "VmFileSystem: Slow refresh failed: Cannot open volume" means a LUN is not ready.
  5. Host logs. /var/run/log/lifecycle.log for download and component errors, /var/run/log/fdm.log for datastore and agent messages.
  6. vCenter logs. For image and depot problems, /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log and imageservice.log.
  7. Paths. Broadcom gives the port matrix for Lifecycle Manager and HA: from vCenter to the host on TCP 80, 443, 902 and 9080 (test with curl -v telnet://<ESXi-Host-IP>:<Port-Number>); from the host to vCenter on TCP 9084 and 9087 (test with nc -zv <VCENTER-IP> <Port-Number>); and between all hosts on 8182 TCP and UDP.

Example (illustrative): cluster cl01 has three hosts, esx01 to esx03. Enabling HA fails on all three with "Applying HA VIBs on the cluster encountered a failure". lifecycle.log on each host repeats Download failed: <urlopen error timed out>. esxcli network firewall ruleset list shows updateManager false on every host. The port tests give 4 checks from vCenter to a host (80, 443, 902, 9080) plus 2 from the host back (9084, 9087), which is 6 per host and 3 × 6 = 18 across the cluster. With the rule enabled, all 18 should succeed. The updateManager line reading false, together with the download timeouts in lifecycle.log, points at the host-side rule.

Fix

Take a snapshot of the vCenter VM before any change to the cluster image or the database. In Enhanced Linked Mode, Broadcom asks for offline snapshots of all vCenter nodes.

Every fix ends the same way: in the vSphere Client select the cluster, go to Configure, vSphere Availability, Edit, disable vSphere HA, wait for the task to finish, then enable it again to force a fresh agent push.

Risk: while HA is switched off the cluster will not restart virtual machines after a host failure. Do it in a quiet window, and re-enable it straight away. To back out, enable HA again.

FDM in the cluster image

Export the cluster image from the cluster's Updates tab (the three-dot menu beside Edit, then Export), remove the vsphere-fdm entry from the components section so it reads "components": null, or lists only the remaining valid components, keep the JSON valid, then Import it back and enable HA. Broadcom's article says FDM is a solution, wrongly listed as a component.

Disruptive step: this changes the cluster image. Keep the original export. To back out, import the original file.

vCenter older than hosts

Disruptive step: upgrading or patching the vCenter appliance takes vCenter offline while it applies. Broadcom's patching article (knowledge base 316584) says to ensure a valid backup of the appliance exists first, with file-based backups recommended, and to create an offline snapshot of the appliance while the vCenter VM is powered off. For a failed patch it says to collect a support log bundle before restoring from a backup or reverting to a snapshot. The back-out is therefore the backup or snapshot, and a revert discards every change made after it was taken.

Upgrade vCenter to a build equal to or higher than the highest ESXi build in the cluster, using that patching article. Then disable and re-enable HA. Broadcom's explanation is that vCenter expects to manage hosts of the same or a lower version, and a host will not accept an older FDM.

Datastore or LUN

Rescan storage on the affected host, confirm df -h lists every datastore, then reconfigure HA. For LUN problems Broadcom lists inconsistent presentation across hosts, a LUN not visible because of zoning or iSCSI settings, a read-only LUN, and a LUN administratively detached. Its fix for a LUN that is no longer used is to unmount and detach it on each host.

Disruptive step: unmounting and detaching a LUN removes that storage from the host. Before you start, confirm nothing uses the LUN. Back out by attaching and mounting the LUN again on each host and rescanning. If anything was using it, the VMs concerned will already have lost that storage.

Update Manager rule

esxcli network firewall ruleset set -e true -r updateManager enables the rule, or tick vCenter Update Manager under the host's Configure tab, Firewall, Edit. Confirm with esxcli network firewall ruleset list.

Risk: this opens a host firewall rule. Undo it with the same command and -e false if it was off by design.

Retired repository URLs

Broadcom says legacy VMware URLs such as hostupdate.vmware.com no longer work for the FDM VIBs. Update the repository settings in Lifecycle Manager to the current Broadcom endpoints and force a fresh HA configuration. Broadcom's note does not print the new URL; take it from Broadcom's current documentation.

Risk: this changes the Lifecycle Manager repository settings, which also affect every other cluster and host that downloads updates through them. Write down the existing URLs first. Back out by putting the old URLs back, which restores the failing state described above.

Stale database entries or damaged agent files

Broadcom's workaround for stale database entries deletes rows from two Lifecycle Manager tables and then recreates the cluster image. It is documented as a known issue with a fix planned for a future release. Broadcom adds that in environments with NSX-T the NSX solution must be re-registered with dcli after the image is recreated, or vSphere Lifecycle Manager may remove NSX-T components.

Disruptive step: database edits and removing the agent VIB from every host are not routine. Snapshot vCenter first, and raise these two cases with Broadcom support rather than working them from the knowledge base alone. For the fdm.cfg case Broadcom gives two options, both with HA disabled. Option 1 is esxcli software vib remove -n vmware-fdm on each host and then enabling HA; it is not applicable from ESX 9.0 onwards. Option 2 is to copy .#fdm.cfg over fdm.cfg in /etc/opt/vmware/fdm and restart the agent with /etc/init.d/vmware-fdm restart, then enable HA. Removing the VIB cannot be undone, though HA reinstalls it when enabled.

Confirm the fix

  • The HA configure task completes on every host.
  • Each host shows the HA agent as running, with no HA agent error on the host or the cluster.
  • esxcli software vib list | grep -i fdm matches the vCenter build.
  • A later Lifecycle Manager compliance check completes.

What to watch afterwards

  • Patch vCenter before ESXi, so vCenter is never older than its hosts.
  • Keep LUN numbers and presentation the same on all hosts.
  • Keep a record of required ports and firewall rules for Lifecycle Manager and HA.
  • Check cluster image contents after upgrades.

Risks and limits

  • These commands have not been run or lab-tested. They come from the Broadcom knowledge base articles listed below.
  • Several separate Broadcom articles carry this first sentence. The table covers seven causes and does not claim to cover every case. Where the logs match none, raise a support case.
  • The database and fdm.cfg workarounds are Broadcom's documented steps, but they are not repeated line by line here on purpose.
  • Menu names and log paths differ by build.

Sources