Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Quiesced snapshot fails: "exceeded the time limit for holding off I/O"

Technical article
VMware

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

The error "Cannot create a quiesced snapshot because the snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine" means the guest operating system could not flush its data and hold its disks still long enough for the snapshot. Broadcom names two causes: I/O in the VM is too high for the flush to finish, or a guest volume has 0 bytes free. Prove which with the checks below, then reduce the I/O during the freeze or fix the failing writer.

What the error means

A quiesced snapshot asks VMware Tools inside the guest to pause applications and flush the file systems before ESXi takes the snapshot. On Windows, Tools does this through the Volume Shadow Copy Service (VSS) and the VMware Snapshot Provider. On Linux, the Tools backup plugin freezes the file systems. If the freeze takes too long, the snapshot is abandoned.

Broadcom's explanation for this exact error (KB 340150): "This issue occurs when the I/O in the virtual machine is high and the quiescing operation is unable to flush all the data to disk, while further I/O is created. This issue also occurs when trying to create a quiesced snapshot on a virtual machine that does not have free space. That is, if the free space on the virtual machine is 0 bytes."

The same fault shows up under other wording. The ones to search for:

  • vSphere Client: "An error occurred while taking a snapshot: Failed to quiesce the virtual machine." (KB 428895)
  • vSphere Client: "An error occurred while quiescing the virtual machine. The error code was: 4 The error was: Quiesce aborted" (KB 344277)
  • The VM's vmware.log: "[msg.snapshot.quiesce.timeout] Timed out while quiescing the virtual machine." (KB 428895)
  • vmware.log: "[msg.snapshot.quiesce.rpc_timeout] A timeout occurred while communicating with VMware Tools in the virtual machine." (KB 415702)
  • vmware.log: "'VssSyncStart' operation failed: IDispatch error #8451 (0x80042303)" (KB 340150)
  • Linux guest, with Tools debug logging on: "SyncDriver: '/db-data' appears locked or frozen by another process. Cannot complete the quiesced snapshot request." and "Canceling backup operation due to timeout." (KB 421195)

KB 428895 notes the typical pattern: the quiesced snapshot fails "while standard (non-quiesced) snapshots succeed without issue". That contrast is the first thing to prove.

Likely causes

Cause What you see How to confirm
Heavy guest I/O during the freeze (database, mail, file server) Fails at busy times, works at quiet times Retry at a quiet time; on Linux, the vmsvc log shows the freeze timing out
A Windows VSS writer failing or timing out "Failed to quiesce", 0x80004004 in the Tools debug log vssadmin list writers shows a writer not in [1] Stable
Windows services in the wrong state, or a third-party VSS provider VSS event IDs 11, 12292, 12032 or 12298 in the Application log vssadmin list providers; service startup types (below)
A guest volume with 0 bytes free The error from KB 340150 with no load pattern Free space on each guest volume
Backup agent installed in the guest without a reboot Started failing straight after an agent install KB 340150: reboot the VM and retry before deeper checks
VMware Tools too old for the host rpc_timeout, "Tools heartbeat timeout" "VMware Tools Not running" on the VM Summary plus rpc_timeout; Tools version 10.1.5 on vSphere 8 (KB 415702)
Linux freeze hang under heavy I/O Freeze reported as timed out after 15 minutes; VM may become unresponsive KB 421195 log lines in /var/log/vmware-vmsvc-root.#.log

Prove the cause before changing anything

1. Take a snapshot without quiescing. KB 344277 says: "Before verifying a VSS quiescing issue, ensure that you are able to create a manual non-quiesced snapshot using the vSphere Snapshot Manager." If the plain snapshot also fails, the fault is not quiescing; look at snapshot and datastore problems instead.

From the host shell, KB 340150 gives the form of the command, with the last two values for memory and quiescing:

vim-cmd vmsvc/snapshot.create 23 "SnapName" "Snap Description" 0 0

Here 23 is the Vmid from vim-cmd vmsvc/getallvms, and 0 0 means no memory and no quiescing. Risk: creating and deleting a snapshot adds disk I/O and a brief stun to a VM that may already be struggling. Run it at a quiet time and delete the snapshot promptly.

2. Read the VM's vmware.log in /vmfs/volumes/<datastore>/<vm_folder>/ (KB 428895). Find the time of the failed snapshot and note which message appears: quiesce.timeout, quiesce.rpc_timeout, or quiesce.vmerr with a VSS error code.

3. Windows: check the writers. In an administrator Command Prompt in the guest, as KB 428895 and KB 344277 show:

vssadmin list writers
vssadmin list providers

Good: every writer shows State: [1] Stable and Last error: No error. Bad: any writer in another state, for example State: [10] Failed with Last error: Timed out, or "Retryable error". KB 428895 adds that such writers "continue to fail even after restarting the VSS service". The providers list should show "Microsoft Software Shadow Copy provider 1.0"; any other provider is a third-party one, which KB 344277 says "may interfere with the quiescing operation".

4. Windows: check the services and event log. KB 344277 gives the expected state during a backup:

  • COM+ System Application: started, Manual. COM+ Event System: started, Automatic.
  • Volume Shadow Copy and Microsoft Software Shadow Copy Provider: Manual.
  • VMware Snapshot Provider: Manual; started during the backup, not running while idle.

In the Application log, look for Event ID 11, 12292, 12032 or 12298, and "The COM Server with CLSID {<GUID>} and name <name> cannot be started." KB 344277 also asks for an accurate, synchronised guest clock.

5. Check guest free space on every volume. A volume at 0 bytes free is one of KB 340150's two named causes.

6. Check the VMware Tools version. KB 415702 says Tools version 10.1.5 "is incompatible with vSphere 8.0 and later versions"; the match is a VM Summary showing "VMware Tools Not running" after an upgrade from vSphere 7 to 8, with the log lines "GuestRpcSendTimedOut: message to toolbox timed out." and "Tools: Tools heartbeat timeout." To read the installed version, KB 344132 gives, on Windows, VMwareToolboxCmd.exe -v run from C:\Program Files\VMware\VMware Tools\; on Linux, vmware-toolbox-cmd -v; and from the host, grep toolbox /vmfs/volumes/datastore/vm_name/vmware.log. 7. Turn on Tools debug logging if the cause is still unclear. KB 313512 (Enabling debug logging for VMware Tools) gives the backup and VSS settings for tools.conf (Windows C:\ProgramData\VMware\VMware Tools\tools.conf, Linux /etc/vmware-tools/tools.conf):

[logging]
log = true
vmbackup.level = debug
vmbackup.handler = vmx
vmvss.level = debug
vmvss.handler = vmx

Use the vmx handler. KB 313512 says that for backup faults "you must be certain not to set the debug logging to a file or file+ handler", because the guest disks are frozen during the operation. It also warns the vmx handler "can have an impact on the performance of the host". KB 313512 says "Restart the VMware Tools service." Disruptive step: the restart briefly interrupts Tools services in the guest. Retry the snapshot and read vmware.log again. To turn logging off, remove the lines: "Logging is not disabled if you change log = true to log = false."

Example (illustrative): sql01 fails its nightly quiesced backup at 02:00 but succeeds when retried at 07:00. SqlServerWriter shows [1] Stable at 07:00, and the 02:00 slot overlaps the database maintenance job. The cause is I/O during the freeze, so the fix is to move one of the two jobs.

Fix

Heavy I/O during the freeze. KB 340150's first option: "Reduce the amount of ongoing I/O to the virtual machine. This is accomplished using pre-freeze scripts or post-freeze scripts to quiesce application I/O." Moving the backup away from batch jobs and other backups does the same.

  • Windows: KB 343375 (A virtual machine can freeze under load) gives an Exchange example: C:\Windows\pre-freeze-script.bat runs net stop MSExchangeSA /yes and C:\Windows\post-thaw-script.bat starts the Exchange services again, taking the service offline "for approximately 60 seconds". The page is written around the SYNC driver era of Windows and does not say where current Windows Tools look for these scripts; confirm the script location in the VMware Tools documentation for your Tools version. Disruptive step: the service is down while the snapshot is taken; test the start-again script first.
  • Linux: KB 421195 says to create /etc/vmware-tools/backupScripts.d/pre-freeze-post-thaw-linux.sh ("If the folder does not exist, create it"). The script "is invoked with a single parameter freeze|thaw|freezeFail", so it can stop the workload on freeze and start it on thaw and freezeFail. KB 453915 adds: make it executable with chmod +x /etc/vmware-tools/backupScripts.d/pre-freeze-post-thaw-linux.sh. Disruptive step: the workload is unavailable until the thaw step runs; handle freezeFail too, or it stays stopped after a failed snapshot.

Linux freeze hang. KB 421195 describes the freeze "timing out after the default timeout of 15 minutes, despite the filesystem being frozen in actuality", which can make the VM unresponsive. Its resolution: "VMware engineering is aware of this issue and actively working towards implementing a fix in a future VMware Tools version." Until then, use the scripts above.

A failing Windows writer. KB 428895: "Engage Microsoft Support to troubleshoot the specific failing VSS writers. Common OS-level remediation includes unregistering and re-registering problematic VSS writers." While that is worked on, KB 325342 shows how to stop Tools calling one writer: list its name, one per line, in vmbackup.conf at %ALLUSERSPROFILE%\Application Data\VMware\VMware Tools\. Risk: the excluded application is no longer application-consistent in the snapshot. Remove the entry once the writer is fixed.

Services, providers and agents. Set the services to the states listed in step 4. KB 344277 says of a third-party provider: "Try uninstalling any third-party VSS providers." If a backup agent was installed without a reboot, KB 340150 says to reboot the VM and retry first. Disruptive step: a guest reboot stops the workload.

Full guest volume. Free space on the volume, or extend it, then retry.

Old VMware Tools. KB 415702: "Upgrade VMware Tools to a version that supports vSphere 8.0 and later releases." Disruptive step: a Tools upgrade can need a guest reboot.

Drop to file-system or crash consistency if application consistency is not needed or cannot be made to work. Two options from Broadcom:

  • KB 326282 (Disabling Application Level Quiescing), Windows only: in C:\ProgramData\VMware\VMware Tools\Tools.conf add [vmbackup] and vss.disableAppQuiescing = true, then restart the VMware Tools service.
  • KB 428895 and KB 326282: with the VM powered off, set the advanced parameter disk.EnableUUID to FALSE, then power on and retry. KB 428895 calls this a way "To workaround or isolate" the fault. Disruptive step: the VM must be off to change it.

KB 340150's other option is a crash-consistent snapshot, with no quiescing. Risk: applications such as databases then rely on their own crash recovery when the backup is restored. Agree this with the application owner.

Confirm the fix

  • A quiesced snapshot completes from the vSphere Client or the backup tool, at the time it used to fail.
  • vmware.log shows no msg.snapshot.quiesce warnings for that snapshot.
  • Windows: vssadmin list writers shows every writer [1] Stable with Last error: No error after the backup.
  • The snapshot has been removed after the backup.

What to watch afterwards

  • Backup timing against database maintenance, indexing, antivirus scans and other backups.
  • VSS writer state after Windows updates and application upgrades.
  • Tools debug logging and log settings. KB 385812 says debug logging and log throttling changes should be "reverted to their default state" after troubleshooting (KB 385812).
  • Excluded writers in vmbackup.conf and any disk.EnableUUID change, so a temporary exclusion does not become permanent.

Risks and limits

  • Broadcom's pages differ on one service: KB 344277 lists the Virtual Disk service with startup type Manual, while KB 385812 says the Microsoft Virtual Disk Service should be set to Automatic. Follow Microsoft's default for the Windows version in use.
  • Version scope differs by page. KB 340150, KB 343375 and KB 421195 cover ESXi 7 and 8, KB 415702 covers vSphere 8, KB 428895 covers 7, 8 and 9, and KB 313512 lists up to 8.0.0. Broadcom documents the other fixes for 7 and 8 only.
  • KB 453915 is written for a Red Hat Enterprise Linux 8 guest running a Broadcom access-management product; it is used here only for the chmod step, which KB 421195's script also needs.

NetLeverage runs its own hosting on VMware vSphere, vCenter and ESXi. Talk to an engineer.

Sources