Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

vSphere Client '503 Service Unavailable': find which vCenter service did not start

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A 503 Service Unavailable from the vSphere Client or the appliance management interface is a server-side error: the web front end is up, but the service behind it is not answering. Sign in to the appliance by SSH and run service-control --status --all. The service listed as stopped, for example vmware-vpxd or vsphere-ui, tells you which log to read. Read that log before you start or restart anything, because the service can stop again for the same reason.

Do not restart everything in a loop. Broadcom documents one case where repeated restarts exhaust the service manager itself.

What the error means

Broadcom's article on the 503 states that it is an HTTP response indicating the server is temporarily unable to handle the request, and that the problem is on the vCenter side, not the browser. The page can show plain "503 Service Unavailable", or a longer form that names a named pipe:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x00007fb7d00200a0] _serverNamespace = / action = Allow _pipeName =/var/run/vmware/vpxd-webserver-pipe)

The pipe name is the useful part. In another Broadcom article the same message appears with service-control --status --all listing vmware-vpxd as stopped. A different form, for the vSphere Client service, names the /ui namespace and port 5090. A third wording, "no healthy upstream", is linked by Broadcom to expired certificates.

Broadcom's list of causes for the 503 is short and mixed: vCenter is down for maintenance, the reverse proxy service is down, the vSphere web client service is down, or firewall settings are wrong. It adds that the vCenter may simply be busy, and advises trying again in a few minutes, from a different client. It also lists the wider reasons services stop: a full disk, expired certificates, a corrupt database (a disconnected database server is listed only for Windows vCenter Server), very high storage latency, very high virtual CPU contention, and time-sync problems between vCenter and its peers.

Likely causes

Cause What you see How to confirm
vCenter still starting, or in maintenance 503 for a short time after a reboot or patch Try again after a few minutes from another browser; service-control --status --all shows services starting
A single service stopped, such as vmware-vpxd 503 naming vpxd-webserver-pipe service-control --status --all lists it as stopped; its log shows why
Appliance disk partition full Several services down together Disk usage on the appliance; follow "vCenter appliance partition full: finding what filled /storage/log, /storage/db or /storage/seat"
Expired certificate 503 or "no healthy upstream"; sign-in errors such as "Signing certificate is not valid" Certificate expiry check; follow "vCenter Sign-In Fails After a Certificate Expires: Find the Store Responsible"
Database index problem vmware-vpxd fails to start; log has "contains unexpected zero page at block" vpxd.log and the Postgres log
Service manager out of file handles service-control fails to start services; "Too many open files" in vmon.log vmon.log after repeated restarts
Firewall settings wrong 503 after a network or firewall change Broadcom lists misconfigured firewall settings as a cause; test the URL from another network

Prove the cause before changing anything

All of these are read-only. Sign in to the appliance as root by SSH. SSH lands in the appliance shell, so type shell to reach the Bash shell before the vmon-cli commands (Broadcom's vmon-cli page says to access the Bash shell). For the database case below, Broadcom's page runs shell.set --enabled true first.

service-control --list
service-control --status --all
service-control --status vmware-vpxd
vmon-cli -l
vmon-cli -s vpxd

Broadcom's service-control article says service-control --status lists and gets the status of services, and a single service reports a line such as "Running: vmware-cm". vmon-cli -l lists the services the service manager watches. vmon-cli -s <name> prints Starttype, RunState, HealthState and the process ID. A healthy vpxd shows RunState: STARTED and HealthState: HEALTHY.

Then read the log of the stopped service. Broadcom's 503 article names vsphere_client_virgo.log in /var/log/vmware/vsphere-client/logs/ (Broadcom's page for vCenter 6.5 prints /var/log/vmware/vsphere-ui/logs instead, so the folder depends on the version) and vpxd.log in /var/log/vmware/vpxd on the appliance. Broadcom's service-manager article names /var/log/vmware/vmon/vmon.log. For the database case, Broadcom names /var/log/vmware/vpostgres/postgres.log.

To preview what a start would do, Broadcom documents a dry run: service-control --start --all --dry-run displays the actions without performing them.

Two other guides cover causes that stop several services at once: "vCenter appliance partition full: finding what filled /storage/log, /storage/db or /storage/seat" and "vCenter Sign-In Fails After a Certificate Expires: Find the Store Responsible". If your findings match either, follow that guide instead.

Example (illustrative): after a power cut, browsers at vcsa01.example.com show the pipe form of the 503. You try three times, five minutes apart: 3 × 5 = 15 minutes. service-control --status --all shows vmware-vpxd stopped and the other services running. vpxd.log ends with a database error naming the index pk_vpx_event_31 and "contains unexpected zero page at block 24". That is the database-index case, not a firewall or a full disk, and restarting vpxd without fixing the index will fail again.

Fix

Take a snapshot of the vCenter VM before any change that touches certificates or the database. For the database fix, Broadcom's page asks for "a snapshot of the VCSA". For the certificate fix, Broadcom's page asks, for Enhanced Linked Mode, for all linked vCenter Servers to be powered off together and a snapshot taken of each node, and, for vCenter HA, for the VCHA configuration to be removed before snapshots are taken.

Start the stopped service

Disruptive step: starting or stopping services interrupts the vSphere Client and API access while it runs. Anything that depends on vCenter, such as backups and automation, will see errors while it runs. To back out, start the services again.

For one service: service-control --start <service name>. For all: service-control --start --all. To restart everything, Broadcom's article says to stop and then start: service-control --stop --all && service-control --start --all. Broadcom states plainly that there is no service-control --restart command.

Broadcom also says that if all services need restarting they must be restarted in a particular order, so it recommends restarting them in the appliance management interface by rebooting the appliance, or using the command-line method above. Do not hand-pick an order.

Database index corruption

Broadcom's fix for the "unexpected zero page" error is to re-index the affected table. Broadcom's command says <table_name>, but the names in its log examples are index names, such as pk_vpx_event_31, so use the index name the log gives.

Disruptive step: this works directly in the vCenter database. Take a snapshot first. A wrong table name or an interrupted run can leave vCenter unusable, and there is no way back other than the snapshot.

The steps in Broadcom's article are: enable the shell with shell.set --enabled true; open the database with /opt/vmware/vpostgres/current/bin/psql -U postgres -d VCDB; run reindex index <index_name>; for each index named in the logs; leave with \q; then service-control --start vmware-vpxd and service-control --start --all. Broadcom's examples are pk_fle_snap_disk_unit, vpx_vm_fle_snap_disk_unit_m1 and pk_vpx_event_31.

Service manager out of file handles

If service-control fails with "An error occurred while starting service" and vmon.log shows "Too many open files", Broadcom's article ties it to a descriptor leak in the service manager. It says each restart increments the open count for vMonTelemetry.db by two, and the fix is in vCenter Server 7.0 Update 3c and later. Recovery on affected builds needs the vCenter appliance to be restarted.

Disruptive step: restarting the appliance takes vCenter offline for the duration. Plan it, and do not keep re-running service-control in the meantime.

Certificates and disk

Follow "vCenter appliance partition full: finding what filled /storage/log, /storage/db or /storage/seat" for a full disk and "vCenter Sign-In Fails After a Certificate Expires: Find the Store Responsible" for certificates. For certificates, Broadcom's 503 article for that case recommends a backup or powered-off snapshot first, and notes that an error during replacement can leave vCenter inaccessible.

Confirm the fix

  • service-control --status --all shows every expected service running.
  • vmon-cli -s vpxd reports RunState: STARTED and HealthState: HEALTHY.
  • The vSphere Client signs in, and the host and VM inventory loads.
  • The same log that explained the stop shows no new error for 30 minutes.

What to watch afterwards

  • Keep headroom on the appliance disk partitions and track certificate expiry dates. Both stop several services at once, and Broadcom lists both as 503 causes.
  • Keep time in step between vCenter, its database and the hosts. Broadcom lists time synchronisation as a cause of services stopping.
  • Take file-based backups of the appliance so that a database problem has a clean recovery route. See "How to back up and rehearse recovery of the vCenter appliance".

Risks and limits

  • These commands have not been run or lab-tested. They are taken from the Broadcom knowledge base articles listed below.
  • Broadcom's 503 article is written for vCenter Server generally and lists Windows paths as well as appliance paths. The paths here are the appliance ones.
  • The vsphere-ui article is for vCenter 6.5 and says the problem was resolved in 6.7. Its workaround is not repeated here.
  • A 503 has many possible causes. The table covers those Broadcom documents, not all of them.
  • Database and certificate repairs can make matters worse if done wrongly. When the log does not match one of these cases, raise a support request instead.

Sources