Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Subscribed content library sync fails: certificate, URL and timeout errors

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A subscribed content library copies templates and files from a published library, usually on another vCenter. When a sync task fails, the reason is almost always in one log on the subscriber vCenter: /var/log/vmware/content-library/cls.log. The documented causes fall into three groups. The publisher's certificate changed or is not trusted ("cannot authenticate SSL certificate", "SSL thumbprint mismatch", "INVALID_ELEMENT_TYPE"). The publisher cannot be reached ("The remote library is not reachable", "Connection refused"). Or the file transfer starts and then fails ("Read timed out", "Connection reset", "Reached the maximum number of retries (10)"). Read the log, match the line, and fix that one cause.

What the error means

When you subscribe, vCenter stores the publisher's certificate thumbprint. The TechDocs page on creating a library: "If prompted, accept the SSL certificate thumbprint. The SSL certificate thumbprint is stored on your system until you delete the subscribed content library from the inventory." If the publisher's certificate is later replaced, the stored thumbprint no longer matches and every sync fails until it is accepted again.

Sync itself runs on a schedule: "the subscribed library automatically synchronizes to the source published library at regular intervals. You can also manually synchronize a single item or an entire subscribed library." KB 427794 places the file transfer between the vCenter Server and the target ESXi host (or the storage provider service doing datastore I/O), so a closed port, a network device or a timeout on that path can break large transfers part-way.

Likely causes

Cause What you see Source
Publisher certificate replaced "cannot authenticate SSL certificate"; cls.log "SSL thumbprint mismatch" KB 422773, KB 411998
Content library extension certificate invalid or expired "INVALID_ELEMENT_TYPE"; "certificate_unknown(46)" KB 323149
Stale STS signing certificates "Access to perform the operation was denied." or INVALID_ELEMENT_TYPE KB 443258
Publisher CA not trusted by subscriber "INTERNAL_SERVER_ERROR" KB 386380
Publisher not reachable on 443 "The remote library is not reachable" KB 323241
Port to the datastore host blocked "Connection refused" KB 427794
NFC session times out on large disks "IO error while uploading to datastore ... Read timed out" KB 378267
Firewall or inspection cuts long transfers "Connection reset" at about the same size each time KB 404495
Too many items syncing at once "Reached the maximum number of retries (10)" KB 344911
Item or datastore removed on the publisher "Invalid response code: 400"; "NFC Error 1/16: No file exists for given path" KB 388148, KB 376974, KB 414730

Prove the cause before changing anything

All of these steps only read.

1. Note the exact task error in the vSphere Client, including any file name and datastore path.

2. Read cls.log on the subscriber vCenter. SSH to the appliance and search /var/log/vmware/content-library/cls.log around the task time. Look for these lines:

  • SSL thumbprint mismatch: Received ..., expected ... and Remote library certificate error: certificate_unknown(46): certificate (KB 422773).
  • errorType = INVALID_ELEMENT_TYPE after recent certificate changes: extension certificate (KB 323149).
  • Transfer failed! item: ...; bytes transferred: ...; total bytes: ...: a transfer cut part-way (KB 404495). Note the bytes figure across attempts.

3. Compare thumbprints. KB 422773 gets the publisher's current SHA-1 fingerprint from the subscriber's shell:

openssl s_client -connect <VC_FQDN>:443 -showcerts </dev/null 2>/dev/null | openssl x509 -noout -fingerprint -sha1

It then compares that with what the subscriber stored, by opening the vCenter database read-only with /opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres and running SELECT * FROM cl_library;. "Compare the thumbprint value in the output against the active thumbprint retrieved in Step 1." Different values confirm a replaced publisher certificate.

4. Test reachability. KB 323241: "Test the DNS name resolution on both Subscriber and Publisher vCenters." Then from each side:

openssl s_client -connect destination-ip:ssl-port
nc -zv <vCenter_ip> 443

Its cause: "Publisher VCenter is not reachable over HTTPS port 443 from Subscriber VCenter". For "Connection refused" on writes, KB 427794 tests ports from the ESXi host: nc -zv X.X.X.X 902, nc -zv X.X.X.X 80 and nc -zv X.X.X.X 443. In its example, TCP 902 returns "No output as Port TCP 902 is disabled."

5. Check whether the failure repeats at the same size. KB 404495 found "the transfer consistently fails when approximately 5 GB of data has been uploaded", which points to "a recurring network connection drop or timeout occurring around the same data threshold". "Bytes transferred can differ from environment to environment."

6. Check the item still exists on the publisher. KB 388148's cause is "Local content library item is unavailable." KB 376974 sees "NFC Error 1/16: No file exists for given path" when files were deleted from the datastore by hand.

Example (illustrative): subscribed library templates-dr stopped syncing the week the publisher vCenter got a new CA-signed certificate. cls.log shows "SSL thumbprint mismatch". The openssl fingerprint differs from the value in cl_library. Port 443 tests succeed both ways. This is KB 422773's case alone.

Fix

Publisher certificate replaced. Accept the new thumbprint in the vSphere Client. KB 422773: "Navigate to Administration > Content Libraries. Locate the impacted subscribed library > Right-click on it and choose Edit Settings or select the Actions menu. Click OK without modifying any of the existing settings." A banner appears: "SSL certificate cannot be trusted. The thumbprint of the certificate is: [thumbprint]. Do you want to proceed?" "Click YES to accept the New Thumbprint and click OK." Check the thumbprint shown matches the openssl output before you accept. KB 411998, written for vCenter Server 8.0.x, documents a database update instead. Risk: its page warns to "take a snapshot or backup of the vCenter Server before making any changes to the vCenter Database"; use the client route first.

Extension certificate (INVALID_ELEMENT_TYPE). KB 323149 exports the vpxd-extension certificate and key, updates the content library extension, and restarts the service:

/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.cl -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <vCenter-FQDN> -u Administrator@<vsphere.local>
service-control --restart vmware-content-library

If Machine SSL or solution user certificates have expired, renew those first. Risk: "Prior to making any changes on the vCenter Server, take a snapshot of the vCenter Virtual Machine. If the vCenter Server is in linked mode, take offline snapshots of all the vCenter Servers in the linked mode". The exported key is a secret: delete the /certificate folder afterwards.

Stale STS certificates. KB 443258, written for vCenter Server 8.0, replaces the STS signing certificate with vCert: Option 3 "Manage Certificates", Option 8 "STS signing certificates", Option 1. Disruptive step: "you must take offline snapshots of ALL vCenter Server and PSC nodes within the SSO domain (ELM group). Modifying the STS identity store can render the environment inoperable if interrupted."

Publisher CA not trusted. KB 386380: export the publisher's root and any intermediate certificates "in base 64 format", build the chain "with the intermediates (if any) on top in order of signing, and the root cert at the bottom", and "add the cert chain to trusted roots under Certificate Management" on the subscriber.

Not reachable or connection refused. Find the blocked path first. KB 323241 runs openssl s_client and nc -zv <vCenter_ip> 443 in both directions between the subscriber and publisher vCenters, and, based on the results, says to "engage Internal Networking team". KB 427794 runs nc -zv X.X.X.X 902, nc -zv X.X.X.X 80 and nc -zv X.X.X.X 443 from the ESXi host shell. Its resolution is "Enable required ports and Sync the Content Library." Risk: opening a port is a firewall rule change; the network team makes it, and the way back is to remove the rule.

Read timed out on large disks. KB 378267 raises two timeouts on the subscriber from 120 to 240 seconds. Back up the file, edit /etc/vmware-content-library/config/ts-config.properties, set transfer.nfcClient.socket.timeout.seconds=240 and transfer.http.request.timeout.seconds=240, then service-control --stop vmware-content-library and service-control --start vmware-content-library. Disruptive step: the content library service stops; running library tasks fail.

Connection reset at the same size. KB 404495: "Engage the internal networking team". It has been seen "with intermediate firewalls or deep packet inspection devices enforcing thresholds on session time or payload size".

Retries exhausted. KB 344911's measures include: "Stagger auto-sync start and stop times for subscriber libraries connected to the same publisher.", "Limit the number of subscriber libraries per publisher.", and reducing how many items sync at once. The TechDocs advanced settings include "Library Maximum Concurrent Sync Items", "Library Auto Sync Start Hour" and "Library Auto Sync Stop Hour". KB 344911 adds: "Usually, a manual sync of the item afterwards will successfully transfer the missing files, provided the Content Library is no longer overwhelmed and has available threads."

Missing items. Re-upload the item on the publisher and re-sync (KB 388148). KB 376974: "Do not remove content library items manually." If the publisher's backing datastore was deleted, KB 414730 says "the subscribed library cannot be converted to a local or publisher library", and to contact Broadcom VCF Support to create a new local or publisher library from the subscribed library's contents, on a "best effort basis".

Confirm the fix

  • Right-click the subscribed library and select Synchronize. The task completes.
  • cls.log shows no new thumbprint, certificate or transfer errors for the task.
  • A large item syncs fully; its size matches the publisher.
  • A test VM deploys from a synced template.

What to watch afterwards

  • Certificate changes on the publisher. Accept the new thumbprint on every subscriber the same day.
  • Sync windows and load across subscribers of one publisher.
  • Firewall session limits on links between sites.
  • Library storage. Delete items through the library, never from the datastore. TechDocs: items deleted from the published library "remain at the storage location of your subscribed library, and you have to manually delete them."

Risks and limits

  • Accepting a thumbprint trusts whatever certificate the publisher presents; compare it with the openssl output first.
  • Certificate and STS changes affect the whole vCenter. Take the snapshots each KB names, offline for linked vCenters where stated.
  • Some "on demand" libraries only fetch metadata during sync; a missing file appears only when the item is used.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources