Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

vCenter update pre-check fails: certificate, ESX Agent Manager and repository errors

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When a vCenter Server update or upgrade stops at the pre-check, the pre-check has found a condition that would break the update. Most of these are trust problems. The documented ones are "VMCA root certificate validation failed", "Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!", "Failed to load the repository manifest data for the configured upgrade", expired certificates behind "Internal error occurs during execution of upgrade process", and weak SHA-1 certificates. Read the exact message and the pre-check log, prove which one applies, take a vCenter backup and snapshot, then fix that one condition and run the update again.

What the error means

The update checks vCenter's own certificates and the certificates of every server it will talk to during the update. The vCenter Server 8.0 Update 3 release notes list the main cases under "Known Issues from Previous Releases":

  • Old root certificate. "If your vCenter system has a legacy VMCA root certificate dating back to version 5.x which does not have the Subject Key Identifier (SKID) extension, upgrades and updates to vCenter 8.0 Update 2 and later fail".
  • ESX Agent Manager (EAM). EAM deploys agent VMs and packages to hosts from download URLs. KB 313026: "In order to harden security, SSL trust pre-checks for the VIB/OVF URLs configured with ESX Agent Manager (EAM) are executed as one of the first steps during the VC upgrade."
  • Custom update repository. "If you use a custom update repository with self-signed certificates that the VMware Certificate Authority (VMCA) does not trust, vCenter Lifecycle Manager fails to download files from such a repository."

Likely causes

Cause What you see How to confirm
VMCA root without SKID "VMCA root certificate does not have 'Subject Key Identifier' extension." Message names it; root carried forward from 5.5 (KB 344774)
EAM URL certificate not trusted "Source ESX Agent Manager Configuration contains URLs that are not trusted by the System!" plus a URL list CollectRequirements EAM log: "is not trusted" (KB 313026)
EAM cannot log in to vCenter (extension certificate mismatch) "Source vSphere ESX Agent Manager (EAM) upgrade failed to obtain EAM URLs" CollectRequirements log: "EAM is still loading from database. Please try again later."; eam.log: "Failed to authenticate extension com.vmware.vim.eam to vCenter" (KB 344775)
Custom repository certificate "Failed to load the repository manifest data for the configured upgrade" Repository combines authentication with a self-signed certificate that vCenter does not trust; KB 313303 states vCenter Server 8.0.0
Expired certificate in VECS "Internal error occurs during execution of upgrade process" PatchRunner.log "certificate has expired"; vecs-cli dates (KB 380569)
Weak signature algorithm "Support for certificates with weak signature algorithms has been removed in vCenter Server 8.0." KB 313460 check script
SHA-1 in a sub-CA chain Upgrade to 8.0 fails on SHA1 chain VMCA is a subordinate CA (KB 399821)
Directory service state "Unable to retrieve the VMDir state of the vCenter." update_microservice.log (KB 443678)

Prove the cause before changing anything

1. Copy the exact message. The pre-check names the failing check. Note the full text, including any URL list.

2. Read the right log on the vCenter appliance (SSH as root):

  • Patching: /var/log/vmware/applmgmt/PatchRunner.log and /var/log/vmware/applmgmt/update_microservice.log (KB 344774, KB 443678).
  • Major upgrade, EAM check: /storage/log/vmware/upgrade/CollectRequirements_com.vmware.eam_<DATE>.log (KB 313026); KB 344775 writes it under /var/log/vmware/upgrade/.

3. Old root certificate. KB 344774: "This issue is caused due to old VMCA Root Certificate without Subject Key Identifier extension. This generally happens if the VMCA Root was carry forwarded from version 5.5 as part of upgrades." A patch to 8.0 U2 can also fail with "Failed to update solution user wcp." in PatchRunner.log.

4. EAM URLs. The log shows each URL tried and "The ('IP/FQDN', 8080) is not trusted." KB 313026 lists three reasons: the certificate's name does not match the host, the certificate is invalid, or "The certificate is not signed by any of the root CA certificates of Photon OS or VECS TRUSTED_ROOTS." If the message instead says EAM could not supply URLs, KB 344775 gives the cause: "an EAM client cannot be created because the EAM service is unable to log in to vCenter. This can occur due to a discrepancy between the 'vpxd-extension' certificate stored in VECS and the certificate information stored in the vCenter Server Database for the EAM extension." On vCenter 7.x and later, eam.log (/var/log/vmware/eam/eam.log) shows [vim:loginExtensionByCertificate:...] Failed. and "Failed to authenticate extension com.vmware.vim.eam to vCenter." 5. Custom repository. KB 313303: "When custom repository is set up with Authentication and Self-signed certificates that are not trusted by VCSA, patcher module does not reach the repository". To see the certificate the repository presents (read-only):

openssl s_client -connect <CustomUpdateRepoFQDN>:<port>

6. Expired certificates. KB 380569's read-only loop prints every store's aliases and expiry dates:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | egrep -i "Alias | Not After" ;done

"Review the output to identify any certificates where the "Not After" date is in the past."

7. Weak algorithms. KB 313460 first checks two vCenter Advanced Settings: "vpxd.ticketing.thumbprintTypes" and "Config.HostAgent.ticketing.thumbprintTypes" (vCenter > Configure > Advanced Settings). Read both values. Risk: KB 313460 says "If the value shows "sha1", change it to "sha256"", which is a change to vCenter's configuration; record the original value first so you can restore it. If that does not clear it, its attached script "verifies certificates across vCenter and all connected ESXi hosts": python /tmp/vsphere8_upgrade_certificate_checks.py. "Resolve any identified "ERROR" or "FAIL" entries before proceeding with the upgrade."

8. Directory service. KB 443678: "The vCenter Server Machine Account password is out of sync." Its log entry pairs the error with "Make sure VMDir state is normal before continuing."

Example (illustrative): vCenter vc-lab was installed as 5.5 and upgraded in place since. The 8.0 U3 update pre-check reports "VMCA root certificate validation failed." That fits KB 344774 alone, so only the root certificate is regenerated.

Fix

Before any certificate change: take a file-based backup and a snapshot of vCenter. KB 399821: "Before proceeding, take snapshot and/or backup of the vCenter VM. If the VCSA is linked (ELM), take offline snapshots of all linked nodes."

Old root certificate. Regenerate the VMCA root with Certificate Manager on the appliance:

/usr/lib/vmware-vmca/bin/certificate-manager

"Select Option 4 or 8": "4. Regenerate a new VMCA Root Certificate and replace all certificates" or "8. Reset all Certificates". KB 344774 also gives the fixcerts route on the appliance: python fixcerts.py replace --certType root, then service-control --stop --all && service-control --start --all. Disruptive step: all vCenter services restart. Risk: "Regenerating Root Certificate will by default replace Machine SSL and Solution User Certificates", so custom certificates must be replaced again. Hosts and integrations that pinned the old certificate need to trust the new one.

EAM URL not trusted. KB 313026 "Option 1 (recommended)": run its attached eam-utility.py install-cert <VIB/OVF/ZIP URL> for each URL, then retry. "The full URL path including the VIB/OVF/ZIP is required." On 8.0 U2a and later, the URL shown ends with a full stop that "has to be removed when running command"; add every URL in the pre-check error. Option 3 is "Replace the SSL certificate with a valid one" and Option 4 is "Add the root CA certificate to VECS TRUSTED_ROOTS." Option 2 disables certificate checking for the URL. Risk: "Using this option makes VIB and OVF URL download insecure." The way back: Option 1 is reverted with eam-utility.py uninstall-cert <VIB/OVF/ZIP URL>, and Option 2 with eam-utility.py enable-trust <VIB/OVF/ZIP URL>. After any option, "the VC upgrade needs to be started again."

EAM cannot log in to vCenter. KB 344775: "This issue has been resolved in vCenter version 8.0 Update 3g." Before the workaround, "Ensure there are valid offline snapshots of the linked vCenter Servers or in case of standalone vCenter Server, take a snapshot without memory." Then update the extension certificates, by either option:

  • Option 1, vCert: ./vCert.py --run config/manage_cert/op_manage-vc-ext-thumbprints.yaml. Press Enter to accept the default SSO administrator account, enter its password, and answer y to "Update extension thumbprints?". Then run service-control --restart vmware-eam.
  • Option 2, the fixcerts script attached to KB 344775: python fixcerts.py update --ExtensionType all. It restarts the eam, rbd and imagebuilder services if they are running.

KB 344775 lists a third option in a separate EAM article. Disruptive step: the EAM service restarts, so EAM is unavailable until it is back; the snapshot is the way back. Then rerun the pre-check. If it still fails, read eam.log and verify that vCenter resolves its own FQDN in DNS.

Custom repository. KB 313303 covers vCenter Server 8.0.0 and a repository that combines authentication with a self-signed certificate that vCenter does not trust. The release notes workaround: "Use CLI, the GUI installer, or the Virtual Appliance Management Interface (VAMI) to perform the upgrade." Or give the repository a trusted certificate, or remove authentication from it. KB 313303 also adds the repository's certificate "at the end of the file /usr/lib/python3.7/site-packages/certifi/cacert.pem". Risk: that edits a system file on the appliance; check the path exists and keep a copy of the original.

Expired certificates. KB 380569 says to "use either of the following tools to renew the expired certificates": vSphere Certificate Manager (/usr/lib/vmware-vmca/bin/certificate-manager) or the vCert tool. With vCert, KB 395406 gives "options 3. Manage certificates -> 2. Solution User certificates" for Solution User certificates, then service-control --stop --all && service-control --start --all. KB 380569 then says: "After the certificates are renewed, take a fresh snapshot of the vCenter virtual machine and proceed with the patching process." Disruptive step: all vCenter services stop and start, so the vSphere Client and API are unavailable until they return. Take the backup and snapshot first.

Weak or SHA-1 certificates. Fix the store the script or the pre-check message names, as KB 313460 sets out:

  • Machine certificate (MACHINE_SSL_CERT). "Replace the vCenter machine SSL certificate with a certificate using a SHA-2 signature. Use the vCert scripted CLI tool or follow the Custom Certificate Authority workflow." Then, for each host, select it in the vSphere Client and click Refresh CA Certificates.
  • Trusted root (TRUSTED_ROOTS). Administration > Certificates > Certificate Management > Trusted Root Store > Add, then upload the SHA-2 certificate. On each host choose Actions > Certificates > Refresh CA Certificates.
  • BACKUP_STORE. The page says the certificate "can be safely removed" using the vCert scripted CLI tool or vecs-cli.
  • ESXi host certificate (rui.crt, in /etc/vmware/ssl/). For VMCA-managed hosts, select the host and choose Actions > Certificates > Renew Certificate. For custom certificates, follow the manual custom-certificate steps in the KB.
  • Hidden certificates on a host. If the script passes but the ESXi upgrade still reports a SHA-1 signature, KB 313460 gives a scan of /etc/vmware/ssl and removal of the named certificate from the store file (for example iofiltervp_castore.pem) after copying it to a .backup file, then /etc/init.d/iofiltervpd restart and a retry of the upgrade. Risk: a service restart on the host.

Disruptive step: each replacement changes a certificate that hosts and integrations trust; hosts need the new CA pushed with Refresh CA Certificates, and dependent solutions need updated leaf certificates. Take the vCenter snapshot first; it is the way back. Where VMCA is a subordinate CA with SHA-1 in its chain, KB 399821 resets the VMCA certificate with vCert or certificate-manager: "All vCenter Server certificates will be regenerated." It then says to reissue the ESXi TLS certificates signed by the previous VMCA certificate, remove the old CA certificates from TRUSTED_ROOTS, and push the change with "Refresh CA Certificates". Disruptive step: every vCenter certificate changes.

Directory service. KB 443678 confirms the directory itself is healthy with /usr/lib/vmware-vmdir/bin/vdcadmintool (Option 6), then: "Reset the vCenter Server machine account password", using the built-in reset_machine_pw.sh script from the KB it links, and restarts all services with service-control --stop --all && service-control --start --all. Disruptive step: all vCenter services stop and start, so the vSphere Client, the API and solutions that use them are unavailable until they return. Take a vCenter snapshot first; KB 443213, which covers a machine-account reset for another fault, says "A valid snapshot of the vCenter Server is required before proceeding."

Confirm the fix

  • Rerun the pre-check: it passes with no certificate, EAM or repository messages.
  • The vecs-cli loop shows no "Not After" date in the past.
  • The update completes and the appliance management interface shows the new version.

What to watch afterwards

  • Certificate expiry dates in every VECS store, well before each update window.
  • Root certificate age on vCenters upgraded in place over many versions.

Risks and limits

  • The KB 313026 and KB 313460 scripts are attachments on those pages; download them from the page, not from another site.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources