Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Hyper-V backup checkpoint fails: Error -0x800423F2, error code 32768 and VSS writer events 8229 and 12293

Technical article
Troubleshooting

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A Hyper-V backup that cannot make its checkpoint reports messages such as "Failed to create VM recovery checkpoint … error code: 32768" or "Could not create backup checkpoint for virtual machine - Error -0x800423F2". The host log may show Event ID 8229 and the guest log Event ID 12293, which Microsoft says "indicate Volume Shadow Copy Service (VSS) writer errors." Microsoft groups the causes into checkpoint and VSS problems, file system and storage problems, permissions and policy, and cluster configuration. Prove which group you are in with read-only checks before you merge a disk file or restart a service.

What the error means

A backup of a running Hyper-V virtual machine starts by asking Hyper-V to make a checkpoint, so the backup reads a stable copy. If the checkpoint cannot be made, the backup stops with an error from the backup software. Microsoft's page lists messages from several products (it names Veeam, Druva, Rubrik and Veritas) and says the codes "include" those above.

Checkpoints matter in a second way. Microsoft's checkpoints page says production checkpoints use "Volume Shadow Copy Service or File System Freeze on a Linux virtual machine to create a data-consistent backup of the virtual machine", while standard checkpoints can "cause data consistency issues with systems that replicate data between different nodes such as Active Directory".

Failed backups also leave things behind. The page describes "Orphaned or unmerged checkpoints: Caused by incomplete backup operations, hidden/invisible checkpoints, or AVHDX chain corruption." Those files take space, and further backups fail on them.

Likely causes

Cause What you see How to confirm
Orphaned or unmerged checkpoints Backups fail; .avhdx files accumulate; Hyper-V Manager shows no matching checkpoint Get-VMSnapshot -VMName against the .avhdx files in the VM folder
VSS writer in a failed or timeout state Events 8229 (host) and 12293 (guest) vssadmin list writers ("Lists all subscribed volume shadow copy writers on the system") shows a writer in a different state from the same writer on a host where backups work
Overlapping backup jobs Failures at the same time each night Compare the schedules of jobs that touch the same host
Missing permissions for the VM accounts "Insufficient permissions" or access errors NT Virtual Machine\Virtual Machines right Log on as a service; icacls on the files
Antivirus filter driver Files in use or blocked during the checkpoint Exclusions against Microsoft's list for Hyper-V hosts
File in use by another process "The process cannot access the file because it is being used by another process. (0x80070020)" Which process holds the file (Process Monitor, Handle, Process Explorer)
Volume low on space Backup fails and VMs may be paused Get-Volume shows little free space

Prove the cause before changing anything

All of these read only.

  1. Write down the exact error text and time from the backup software, and the VM name.
  2. Check the logs for the events Microsoft names. The page does not say which log holds each one, so search both:
Get-WinEvent -FilterHashtable @{ LogName='Application'; Id=8229,12293 }
Get-WinEvent -FilterHashtable @{ LogName='System'; Id=8229,12293 }

Run both queries on the host, and then run both again inside the virtual machine, because Event ID 12293 is logged in the guest. Where a command below writes to C:\Temp, create that folder first if it does not exist.

  1. List the writers and providers:
vssadmin list writers
vssadmin list providers

Save the writer output to a file, for example by adding > C:\Temp\writers-before.txt, and note any writer whose entry differs from the same writer on a host where backups work.

  1. List the VM's checkpoints and the disks it is using:
Get-VMSnapshot -VMName VMName
Get-VMHardDiskDrive -VMName VMName

Then look in the VM's disk folder for .avhdx files that are neither listed as checkpoints nor attached to the VM. Microsoft says to "Inspect the VM disk folder for AVHDX files not listed in Hyper-V Manager." A file the VM is currently using is not an orphan; do not treat it as one.

  1. Check free space on the volume with Get-Volume.
  2. Compare the backup schedules for overlap, and check the host's antivirus against Microsoft's exclusions page.

Example (illustrative): on hv01 at contoso.com.au, backup job A starts at 01:00 and runs 25 minutes, ending at 01:25. Job B starts at 01:10 and runs 25 minutes, ending at 01:35. The two overlap from 01:10 to 01:25, which is 15 minutes, and both ask VSS for a checkpoint at the same time. Moving job B to 01:30 removes the overlap, because job A ends at 01:25.

Fix

Start with the lightest change that matches what you proved.

Stagger overlapping jobs

Microsoft lists "Stagger backup schedules to avoid overlapping VSS operations." Save first: record the old schedule. Risk: changing the schedule below moves a recovery point. Change the schedule in the backup software so no two jobs touch the same host at the same time. To undo it: restore the old schedule.

Restart the VSS services

First save the vssadmin list writers output (step 3 above) and check that no backup or shadow-copy job is running.

Disruptive step: stopping VSS while a backup is running fails that backup. Impact: any job in progress on the host stops and must be rerun. Back-out: if a service does not start, run its net start line again; net start is the reverse of each net stop.

Microsoft's sequence is:

net stop vss
net start vss
net stop swprv
net start swprv

Then list the writers again. Microsoft says to "resolve failed writers by restarting related services or rebooting the host." Disruptive step: rebooting the host stops every VM on it. Move or shut down the VMs first by your normal maintenance method. There is no back-out for the reboot itself.

Merge orphaned checkpoint files

This is the step with the most risk. Microsoft's note: "Always back up critical VMs before performing file merges, deletions, or major configuration changes."

  1. Take a backup you have tested, or export the VM:
Export-VM -Name VMName -Path D:\Export

An export covers the files the VM has attached; it does not copy an orphaned .avhdx, so step 3 also copies the files themselves.

  1. Record which disk files the VM uses (Get-VMHardDiskDrive), and confirm which .avhdx is an orphan using the checks above.
  2. Shut down the VM, then copy the whole VM folder, including the .vhdx and every .avhdx in it, and any disk the step 2 record shows outside that folder, to another volume that has room (replace the paths with your own):
Copy-Item -Path "E:\VMs\VMName" -Destination "D:\Export\VMName-files" -Recurse

Step 4 is the merge. Read its mark first.

Disruptive step: a merge writes the changes in the .avhdx into its parent disk and cannot be undone. Impact: the VM must point to the merged disk, and anything that depended on the unmerged file is changed. Back-out: with the VM shut down, copy the saved files from D:\Export\VMName-files back over the VM folder, replacing the merged ones, for example:

Copy-Item -Path "D:\Export\VMName-files\*" -Destination "E:\VMs\VMName" -Recurse -Force

If you took a backup instead, restore it with your backup product. If you took an export, Microsoft's Import-VM page says -Copy -GenerateNewId copies the files and gives the imported machine a new unique identifier, so it can sit beside the original: Import-VM -Path 'D:\Export\VMName\Virtual Machines\VM-ID.vmcx' -Copy -GenerateNewId, where VM-ID.vmcx is the file name in the export's Virtual Machines folder. An imported copy holds only what the export held.

  1. Merge with Hyper-V Manager. Microsoft's list: select the shut-down VM, then Actions, Edit Disk; on the Locate Disk tab browse to the .avhdx file and select Next; on Choose Action select Merge, then To the parent virtual hard disk, and complete the wizard.

Microsoft's page also lists a PowerShell merge with Merge-VHD, which is not a step here: Merge-VHD needs the exact child disk as -DestinationPath, and the cmdlet page says the child disks "are not deleted, but are no longer valid virtual hard disk files after the operation". A wrong choice cannot be undone, so use the wizard.

  1. Reattach and start the VM. The page: "Verify the VM points to the correct (merged) VHDX file. Start the VM and confirm functionality."

Permissions, policy and antivirus

The page's corrections are: "Ensure the NT Virtual Machine\Virtual Machines account has Log on as a Service rights via Group Policy or Local Security Policy", "Use icacls to verify and grant Full Control to VM service accounts", and "Configure antivirus exclusions by following the Microsoft antivirus exclusions for Hyper-V hosts."

Risk: a Group Policy change reaches every server in its scope, and an exclusion reduces scanning for the excluded paths. Record the current policy value and exclusion list first; the back-out is to restore them. Before an icacls grant, save the folder's permissions with icacls "path\*" /save C:\Temp\acl.txt /t, which icacls path /restore C:\Temp\acl.txt reverses (the folder path without a trailing backslash; if it contains spaces, put quotes around it, because a backslash directly before a closing quote can be read as an escaped quote).

Vendor leftovers

For checkpoints made by a backup product, the page says to "Use vendor-specific tools" to clear stuck checkpoints, and gives one example for one vendor. Use your backup vendor's own tool and documentation for that; do not delete checkpoint files by hand.

Confirm the fix

  • vssadmin list writers shows every writer as it appears on a host where backups work.
  • Get-VMSnapshot lists no checkpoint left by the failed job, and the VM folder holds no unexplained .avhdx files.
  • A manual backup of the VM completes, and then the scheduled job runs without overlap.
  • Events 8229 and 12293 do not return in the next run.
  • The restore check passes: restore the backup to a spare location and open it.

What to watch afterwards

  • Backup duration, so schedules do not drift back into overlap.
  • New .avhdx files that are not tied to a checkpoint.
  • Free space on the volume, since orphaned files fill it.
  • Maintenance practice. Microsoft's list includes "Rebooting cluster nodes without draining roles or placing them in maintenance mode" as an improper practice.

Risks and limits

These commands have not been run or lab-tested.

  • Microsoft's page is a broad checklist. It does not map each error code to one cause, so work from the checks, not from the code alone.
  • The page also lists file-system repair (chkdsk, Repair-Volume) and deleting .vmrs and .vmgs files for a VM stuck in a Saved state. Those change the file system or discard saved state, so they are not steps here; diagnosing a backup-checkpoint fault does not need them.
  • A merge cannot be undone. Without a tested export or backup, do not start.
  • Error text differs by backup product. Microsoft's page names messages from Veeam, Druva, Rubrik and Veritas software; yours may read differently.
  • Check the current Microsoft pages before relying on a command or event ID.

NetLeverage includes a daily full disaster-recovery backup of each hosted server at no extra charge. Talk to an engineer.

Sources