Australian cloud for business software — built and run in Sydney since 2003

+61 2 8282 5656
support@netleverage.com
Talk to an engineer
Talk to an engineer
NetLeverage
NetLeverage

My VM freezes for a while when a snapshot is created or removed

Troubleshooting article
VMwareBackup & RecoveryPerformanceTroubleshooting

A VM drops pings, a remote console session disconnects, or an application times out at the moment a snapshot is created or deleted, for example while your backup product runs. The VM comes back by itself, but the pause was long enough for users or monitoring to notice.

That pause is called a stun. A stun is a short period in which the host stops the virtual machine so it can safely change something that cannot change while the guest runs, such as the set of disk files behind the VM. The fault is a stun that lasts long enough to hurt.

This article shows how to read stun lengths from the VM's log, what the cited Broadcom pages say makes them long, and how to assess changes against application and recovery requirements. The "How to prove it" section only reads.

What is actually happening

A snapshot freezes the VM's disks at a point in time and sends new writes to a separate delta file. Creating a snapshot switches the VM to new delta files. Removing a snapshot merges a delta back into its parent. This is the general mechanism; the pages below describe the stun it causes.

For creation, KB 337998 states that the stun time directly correlates with the number of virtual disks, and that total stun is the single-disk stun multiplied by the number of disks. The page covers creation only.

For removal, KB 323397 describes the host merging the snapshot using a temporary "Consolidate Helper" delta, with the VM stunned for the duration of the consolidation. It says this normally takes almost no time, but a VM with a large amount of delta in the temporary snapshot can be stunned long enough to disrupt applications. The host estimates the stun in advance. If the estimate is over fifteen seconds it waits for another round, up to ten rounds. The page notes the behaviour has changed: if it detects over fifteen seconds after the ninth round, it stops trying and cannot delete the snapshot. Check the page for the behaviour on your build.

Memory snapshots are described differently by two pages. KB 342618 states that taking a memory snapshot stuns the entire state of the VM. KB 321376 states that the host writes the VM's memory to disk, and that the VM can be unresponsive meanwhile.

Create:           stun while every disk is switched (grows with disk count, KB 337998)
Remove:           merge runs, final commit stuns the VM (KB 323397)
Memory snapshot:  stun (KB 342618), then slow or unreachable while memory is written (KB 321376)

How to prove it

The log of each VM is vmware.log in the VM's home folder on the datastore. The pages we cite show stun values in these lines, and the unit is microseconds. KB 374503 prints:

Checkpoint_Unstun: vm stopped for 2687873 us
CPT: vm was stunned for 2806449 us

Divide by one million for seconds: the first value is about 2.7 seconds. The two values on that page differ, so record which one you use. KB 344721 prints the Checkpoint_Unstun form (a value of 63104614 us, about 63.1 seconds). KB 337998 ties the Checkpoint_Unstun value to snapshot creation. No fetched page shows this line for snapshot removal, so for removal use step 4, and treat any Checkpoint_Unstun line you find near a removal as your own observation.

The commands and paths below are plain shell and general vSphere practice, not taken from a Broadcom page.

  1. Note the exact time of the symptom from monitoring or the backup job log.
  2. Use existing approved read access on the host that runs the VM, then change to the VM folder. Enabling SSH is a separate configuration change requiring an access and restoration plan; it is not part of these read-only checks. Quote the path, because folder names often contain spaces:
cd "/vmfs/volumes/<datastore>/<vm folder>"
ls vmware*.log
  1. List the stun lines in the current and rotated logs. With more than one file the file name is printed too:
grep Checkpoint_Unstun vmware*.log

Confirm the log time zone against monitoring before correlating events. Even a subsecond pause may matter to an application; a longer pause coinciding with an error is evidence to investigate, not proof of causation. Compare measured pauses with application logs and the agreed response threshold.

grep SnapshotVMXNeedConsolidateIteration vmware*.log

If the lines are present around the symptom time, a removal was in progress and the estimate shows how much remained to commit. If they are absent, no page we cite lets you identify the operation from the log. Use the operation time in the vCenter task and event history instead (general practice). 5. Count the VM's virtual disks in the vSphere Client under Edit Settings. 6. Check the guest side. KB 337998 lists remote console disconnects and ping drops during snapshot creation, and KB 323397 says a remote session may drop during a long snapshot. Compare guest application and system log times with the log times.

If no log line matches the symptom time, the available evidence does not establish a stun; log retention and operation-specific logging may limit the result. Investigate storage latency and quiescing alongside application evidence rather than ruling stun out.

Causes the pages name

Many virtual disks, at creation. KB 337998 gives an example VM with 249 disks, a stated unstun value of 0.55 seconds, and a logged Checkpoint_Unstun of 132490356 microseconds, about 132.5 seconds. The page's own arithmetic does not match: 249 multiplied by 0.55 seconds is about 137 seconds, while the logged value is 132.5 seconds. Treat the multiplication as an estimate, not an exact rule. Evidence: the same VM shows shorter stuns with fewer disks.

A large final delta at removal. KB 323397 names a large volume of I/O that must be written back as a main cause, and says busy VMs consolidate for longer and produce a larger helper delta. Evidence: the SnapshotVMXNeedConsolidateIteration lines show a large helper before the stun.

Slow storage. KB 323397 names underlying storage latency as the other main cause for removal. Evidence: high device latency at the time, and long stuns on several VMs on one datastore.

Memory snapshots. KB 321376 treats the pause as expected behaviour with no fix, and says its length depends on the memory to be written, how fast that memory changes, and the speed of the backing storage. Evidence: the snapshot was taken with the memory option.

Quiescing under heavy load. Quiescing asks the guest to flush and hold writes while the snapshot is cut. KB 343375 states that an I/O-heavy VM can freeze during a quiesced snapshot, that the delay can become too long, and gives Exchange as its example. The page's sync driver is, in its words, only required for legacy versions of Windows, and newer VMware Tools remove it; on newer Windows the quiescing is done through VSS, which KB 344277 covers per the page. KB 342618 states that quiescing and memory snapshots are mutually exclusive.

A storage-type fault. KB 374503 documents long snapshot stuns on datastores backed by NVMe over TCP. It states a fix in ESXi 8.0 Update 3b, a fix that "will be" delivered in ESXi 7.0.3 P10 (check the page for current status), and that ESXi 9.0 is not affected. KB 344721 documents a stun of about 63.1 seconds on virtual volume datastores when another host held a bind on the snapshot volume; the fix was to put the VM and the backup proxy VM on the same host. That case is documented for vCenter and ESXi 6.5 and 6.7 only, and the page names no later release.

Recovery decisions and change planning

Observation, disruptive configuration changes and disk-chain changes have different consequences. None of the options below is a tested production procedure for this article.

First correlate measured log values, operation timestamps and application symptoms. Establish an application-specific acceptable pause and preserve a baseline. A helper-disk estimate is not a measured outage, and total task duration is not the stun duration.

Inspect snapshot age and backup cleanup without deleting anything as a diagnostic test. Any intended deletion or consolidation needs the exact chain and target verified, a demonstrated restore route, capacity assessment, agreement about lost restore points and an impact window. A running merge must not be interrupted.

With the backup supplier, assess whether snapshot scheduling or memory capture can change without compromising required recovery semantics. Do not prescribe disabling memory capture for every product or workload. Review the effect on backup consistency and recovery acceptance before changing policy.

Storage migration, fewer disks per VM and alternative backup designs are separate architecture changes. Assess capacity, dependencies, rollback and application correctness in a disposable test first. A workload-specific quiescing workaround that stops an application service requires an explicitly accepted outage and restart/health checks; do not apply a legacy workaround to a newer build without current vendor confirmation.

For build-specific NVMe/TCP or vVol cases, match the source's stated versions and symptoms. The ESXi 6.5/6.7 case does not establish a remedy for newer releases. After any approved change, compare observed pauses and end-to-end application behaviour against the baseline, then demonstrate that backup and restore still meet the required protection semantics.

What not to do

  • Do not cancel a snapshot deletion or consolidation because the VM looks frozen. KB 318825 states cancelling can corrupt data, and KB 338534 says consolidation can no longer be stopped once started.
  • Do not edit delta or descriptor files to make a snapshot go away. KB 342618 warns that manual changes to child disks or snapshot configuration may compromise the disk chain.
  • Do not treat snapshots as backups. KB 318825 and KB 342618 both say so, and a longer-lived snapshot gives the removal more to handle.
  • Do not disable the legacy sync driver on a guest that needs application consistency. KB 343375 states the result is crash-consistent snapshots only.
  • Do not read the vCenter task time as the stun. KB 323397 describes the merge running while the VM runs and only the final commit stunning it, so the task lasts longer than the stun.
  • Do not run a consolidation on a busy production VM at peak time to see how long it takes. KB 338534 calls consolidation very I/O intensive and says it can affect other VMs on the datastore.

Acceptance checklist

These are required checks, not completed test results.

  • [ ] Existing approved access used; log/monitoring times reconciled.
  • [ ] Measured stun separated from estimates and task duration.
  • [ ] Application-specific pause threshold and baseline established.
  • [ ] Each proposed change has scope, impact, recovery and version review.
  • [ ] Snapshot removal and service outage require separate approval.
  • [ ] Application performance and backup/restore semantics checked after change.

Sources

  1. KB 337998, "VM snapshot stun times correlate with the number of virtual disks", https://knowledge.broadcom.com/external/article/337998 (ESXi 8.x, 7.x, 6.x).
  2. KB 323397, "Snapshot Removal Stops a Virtual Machine for a Long Time", https://knowledge.broadcom.com/external/article/323397 (ESXi 8.0, 7.0, 6.7, 6.5, 6.0).
  3. KB 318825, "Best practices for using VMware snapshots in the vSphere environment", https://knowledge.broadcom.com/external/article/318825 (ESXi 8.x and 9.x, vCenter Server 8.x and 9.x).
  4. KB 342618, "Overview of virtual machine snapshots in vSphere", https://knowledge.broadcom.com/external/article/342618 (vCenter Server and ESXi 7.x, 8.x, 9.x).
  5. KB 338534, "Estimate the time required to consolidate virtual machine snapshots", https://knowledge.broadcom.com/external/article/338534 (ESXi 7.x and 8.x).
  6. KB 321376, "Virtual machine becomes unresponsive or inactive when taking memory snapshot", https://knowledge.broadcom.com/external/article/321376 (ESXi 7.x and 8.x).
  7. KB 343375, "A virtual machine can freeze under load when taking quiesced snapshots or use custom quiescing scripts", https://knowledge.broadcom.com/external/article/343375 (ESXi 5.x to 8.x).
  8. KB 374503, "A long snapshot stun time may occur when virtual machine runs on a datastore backed by NVMe-TCP", https://knowledge.broadcom.com/external/article/374503 (ESXi 7.x, 8.x, 9.0 as stated on the page).
  9. KB 344721, "Long stun time while backing up virtual machine located in vVOL Datastores", https://knowledge.broadcom.com/external/article/344721 (vCenter and ESXi 6.5 and 6.7).