Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

vSAN objects degraded and the rebuild does not start or stalls

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

After a host, disk or network failure, vSAN rebuilds the missing copies of each object on other hosts. Sometimes it does not. The vSAN object health check shows "Reduced availability with no rebuild", and the resync view is empty or barely moves. KB 385514 sets out the stakes: "vSAN objects will still be accessible. However, object data is at risk if additional failures occur." Read the exact state shown before you touch anything. Some states mean vSAN is waiting on purpose. Others mean it is blocked, and the fix depends on what blocks it.

What the error means

The states come from the "Data Health – vSAN Object Health" check. KB 326929 defines them:

State What vSAN is doing
Reduced availability with no rebuild - delay timer Waiting "for the 60-minute (default) delay timer to expire before issuing the re-protect"
Reduced availability with no rebuild Not rebuilding, "not due to the delay timer ... but due to other reasons"
Reduced Availability With Paused Rebuild "the object rebuild is paused because of lack of available resources"

For the plain "no rebuild" state, KB 326929 adds: "This could be because there are not enough resources in the cluster, or this could be because there was not enough resources in the past, or there was a failure to re-protect in the past and VSAN has yet to retry. Refer to the limits health check for a first assessment if any resources may be exhausted."

On the command line the state reads reduced-availability-with-no-rebuild (KB 416613).

Likely causes

Cause Sign Source
Repair delay timer still running "delay timer" state; host failed under 60 minutes ago KB 327031, 326929
Not enough fault domains or hosts for the policy "no rebuild"; too few hosts left KB 385514
Not enough free capacity "no rebuild" or "paused"; on OSA, clomd.log "no free space" KB 385514, 413125
One host reports every object as reduced availability while the other hosts report healthy Same command shows different results on different hosts KB 385514
Object owned by the host in maintenance mode "Repair object immediately" has no effect KB 416613
Resync throttle set Resync running but slow KB 445909
Network or partition problem Hosts missing from cluster membership KB 445909, 385514

The timer is the first thing to rule out. KB 327031: "By default, the repair delay value is set to 60 minutes. In the event of a host failure, vSAN waits 60 minutes before rebuilding any vSAN objects with components located on that particular host. This is because vSAN is not certain if the failure is transient or permanent." It does not apply to every failure: "If a failure in a physical hardware component is detected, such as a cache or capacity disk, vSAN immediately responds by rebuilding a disk object."

Prove the cause before changing anything

These commands only read. Run them from the ESXi shell of a host in the cluster.

1. Get the health summary on each host. KB 385514 and KB 445909:

esxcli vsan debug object health summary get

Compare the result across hosts. If one host reports all objects as reduced availability with no rebuild while the remaining hosts show healthy objects, KB 385514 treats it as a problem on that host. See the host fix below.

2. List the affected objects and their owner. KB 416613:

esxcli vsan debug object list --health=reduced-availability-with-no-rebuild

Look at the Owner: line. If every stuck object is owned by the host now in maintenance mode, KB 416613's cause applies.

3. Check the repair timer and the throttle. KB 445909:

esxcli system settings advanced list -o /VSAN/ClomRepairDelay
esxcli vsan resync throttle get

"A throttle value of 0 means no manual resync throttle is configured."

4. Watch resync progress over time. KB 445909:

esxcli vsan debug resync summary get
esxcli vsan debug resync list
esxcli vsan resync bandwidth get

KB 445909: "Take multiple samples 30 to 60 minutes apart." It warns: "The “GB left to resync” value can fluctuate. An increase does not always mean the resync failed."

5. Count hosts against the policy. KB 385514 lists the minimums: "FTT=1 RAID-1 requires minimum 3 fault domain or 3 standalone hosts", "FTT=1 RAID-5 requires minimum 4 fault domain or 4 standalone hosts", "FTT=2 RAID-1 requires minimum 5 fault domain or 5 standalone hosts", "FTT=2 RAID-6 requires minimum 6 fault domain or 6 standalone hosts". With one host down, does the cluster still meet the minimum for each policy in use?

6. Check capacity. Compare datastore use with the host rebuild threshold in the vSAN capacity view. KB 385514: "If the used space exceeds this threshold and a host fails, vSAN may be unable to restore all data from the failed host." On vSAN OSA, /var/run/log/clomd.log shows the capacity failure. KB 413125 (vSAN 7.x, 8.x and 9.x OSA) shows CLOMReconfigure: WorkItem failed ignoring disk capacity/transient capacity/component limit: Not found. and CLOM_PublishResyncBytes: About to fail the workItem(Underlying device has no free space), with capFull in the Disk dropped due to line. KB 413125 covers OSA only. For ESA, rely on the capacity view and the host rebuild threshold.

7. Check membership and network. KB 445909: esxcli vsan cluster get, esxcli vsan network list, esxcli vsan cluster unicastagent list and vmkping -I vmkX <peer-vSAN-vmk-IP>.

Example (illustrative): a four-host cluster uses an FTT=1 RAID-5 policy. One host fails with a motherboard fault. After 60 minutes the state moves from "delay timer" to plain "no rebuild" and nothing resyncs. Only three hosts remain, and KB 385514 says RAID-5 at FTT=1 needs four. vSAN has nowhere valid to place the rebuilt component. The fix is to bring the fourth host back or add one, not to change timers.

Fix

Timer still running, failed host coming back. Wait. KB 326929: "if you know that the failed host is actively rebooting or knows that a wrong drive is incorrectly pulled and it is being reinserted, then it is advisable to just wait for those tasks to finish".

Timer still running, host will not come back in time. KB 326929: "You can choose to issue an explicit request to skip the delay timer and start re-protect immediately, if it is known that the failed entity cannot be recovered within the delay period." KB 327031 names the "Repair objects immediately" button in the vSAN Health plug-in and describes it for vSAN 7.x. KB 416613 shows the same action as the "Repair object immediately" option in Cluster > Monitor > vSAN > Resyncing Objects. Neither KB describes a different path for vSAN 8 or 9. Risk: if the host returns after the rebuild starts, the cluster will have done a full resync it did not need.

Changing the timer. KB 327031: "navigate to Cluster > Configure > vSAN > Services > Advanced Options" and "Change Object Repair Timer Option to the desired value". "You must wait for at least 180 seconds before proceeding with any Maintenance/Disk Group/Disk/reboot-related activity after making the above changes through the vSphere UI". KB 445909 warns: "Do not reduce the repair delay timer as a method to speed up an active resync. Once vSAN repair or resync is already in progress, changing the timer does not make the current resync complete faster." Risk: KB 327031: "setting the above option too aggressively can cause unnecessary resync operations to occur". Consider ESXi updates, host boot time including power-on self test, and SSD log recovery before shortening it. Reset it afterwards. KB 327031: "After the failure has been resolved, the setting should be reset to the default of 60 minutes." Way back, per host (KB 445909): esxcli system settings advanced set -o /VSAN/ClomRepairDelay -i 60.

Not enough hosts or capacity. KB 385514: "Rebuild operations cannot proceed because of resource constraints such as insufficient capacity or an inadequate number of fault domains." Return the failed host, add a host or add capacity. Before adding capacity, confirm the host minimum for each policy in use (step 5). Resyncs will follow when a host or capacity is added. Keep free space up: "Maintain adequate free capacity in the vSAN datastore, with a 30% free capacity being the recommended threshold."

Owner is the host in maintenance mode. KB 416613 runs vsish -e set /vmkModules/vsan/dom/ownerAbdicate <Affected_Object_UUID>, which "will force the object to refresh the ownership to another host in the vSAN Cluster". Risk: this is an advanced vsish write from a KB, not part of the standard esxcli set. It forces an owner change on the live object. KB 416613 gives no way back for it. Run it only per affected object, as KB 416613 shows, and only after step 2 confirms the owner.

One host reports every object as reduced availability. If the other hosts show healthy objects, KB 385514: perform a datastore evacuation precheck on the affected host "to verify that placing it in maintenance mode would not render any objects inaccessible", place the host in maintenance mode using the Ensure Accessibility option, then reboot the host. Disruptive step: this is a host reboot. KB 385514: "Before rebooting a host make sure all object in vSAN cluster is in healthy state and accessible." Way back: exit maintenance mode when the host is up.

Resync throttled. Remove the cap with esxcli vsan resync throttle set --level=0 (KB 445909). The KB's own note: "Applying a throttle can reduce the impact on running workloads, but it can also extend the time that objects remain reduced redundancy or non-compliant." Risk: removing the cap can raise VM latency while the resync runs.

Network or partition. Fix membership first. A rebuild cannot finish across a split cluster. The related article on recording vSAN partition membership covers that work.

Confirm the fix

  • esxcli vsan debug object health summary get shows no reduced-availability objects.
  • The resync summary reaches zero across several samples.
  • vSAN object health is green in the vSphere Client.
  • The repair timer is back to 60 minutes, and the throttle is at the level you intend.

What to watch afterwards

  • Free capacity against the host rebuild threshold.
  • Policy changes to higher FTT or RAID-5/6 on small clusters.
  • Maintenance mode choices. KB 416613's case followed maintenance mode on one host.
  • Timer and throttle settings left changed after an incident.

Risks and limits

  • KB 445909: "debug commands and output can vary by release".
  • KB 327031's older per-host method restarts clomd. It notes that "if a virtual machine is being provisioned at the time the clomd service is restarted, that provisioning task may fail."
  • Before rebooting a host during recovery, KB 385514: "Before rebooting a host make sure all object in vSAN cluster is in healthy state and accessible."

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources