Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Windows Data Deduplication: deleted files but the volume is still full

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When you delete files on a volume that uses Data Deduplication, the free space does not come back straight away. The chunks those files used stay in the chunk store until a Garbage Collection job finds that nothing references them and compacts the containers that hold them. By default that job runs once a week, and the thorough "full" form runs only about every fourth time.

So the first question is not "what is using the space?" but "has Garbage Collection run since I deleted the files, did it succeed, and is a full pass allowed on this volume?" The checks below answer that without changing anything.

What the symptom means

Microsoft describes Data Deduplication as post-processing. The Optimization job chunks files, stores unique chunks in the chunk store, and "replace[s] the original file stream of now optimized files with a reparse point to the chunk store". The chunk store is a series of container files in the System Volume Information folder. For an optimised file, then, the space it occupies lives in the chunk store, not in the file.

Deleting such a file removes the reference. The chunks stay. The Garbage Collection job "reclaims disk space by removing unnecessary chunks that are no longer being referenced by files that have been recently modified or deleted". Its default schedule is "Every Saturday at 2:35 AM".

A useful test follows from this. A file that was never optimised has no chunks to collect, so deleting it should free space at once. A file that was optimised frees nothing until Garbage Collection runs. The Default usage type does not optimise files younger than 3 days (the MinimumFileAgeDays setting) and does not optimise files in use.

Regular and full Garbage Collection

Microsoft documents two kinds:

  • Regular. It "uses a statistical algorithm to find large unreferenced chunks" and "compacts a chunk store container only if a minimum percentage of the chunks is unreferenced". It runs "once a week" by default. The percentage is not published on these pages.
  • Full. It "compacts every container even if just a single chunk in the container is unreferenced" and "will also free space that may have been in use if there was a crash or power failure during an Optimization job". By default "every fourth job is a full Garbage Collection job". Microsoft says a full pass "will typically find and release up to 5 percent more of the unreferenced data than a regular Garbage Collection job".

That last figure is a useful yardstick. If a successful regular job ran and far more than a few percent of the deleted data is still held, a missing full pass is probably not the whole story.

Full Garbage Collection is "always disabled (regardless of the registry value) for volumes with the Backup Usage Type", and the DeepGCInterval registry setting can turn it off on any volume. Microsoft gives a reason to do so: "Garbage Collection could adversely affect the volume's lifetime shadow copies and the size of incremental backup." Disabling it may be a deliberate trade between space and backup size.

Likely causes

Cause What you see How to confirm
Garbage Collection has not run since the delete Free space unchanged for days after a large delete LastGarbageCollectionTime is older than the delete
The Garbage Collection schedule was removed or disabled No Saturday run, ever Get-DedupSchedule -Type GarbageCollection returns nothing, or shows it disabled
The last job failed Time is recent but result is not success LastGarbageCollectionResult is not 0; message and Event Viewer log say why
Only regular jobs ran Most space returned, a remainder stays Several successful runs; no recent full pass
Full pass disabled The remainder never returns by schedule Backup usage type, or DeepGCInterval set to disable
A job is queued or running Manual job seems to do nothing Get-DedupJob shows the volume busy
Optimization is behind on new data Volume full of new files, savings rate falling LastOptimizationTime and SavingsRate trend

Prove the cause before changing anything

These commands only read. Start PowerShell as administrator, which Microsoft's cmdlet pages require.

Get-DedupStatus -Volume "D:" | Format-List -Property *
Get-DedupJob -Volume "D:"
Get-DedupSchedule -Type GarbageCollection
Get-DedupVolume -Volume "D:" | Format-List -Property *

Microsoft's Running Data Deduplication page names the fields to read in the Get-DedupStatus output: for Garbage Collection, "LastGarbageCollectionResult (0 = Success), LastGarbageCollectionResultMessage, and LastGarbageCollectionTime (should be recent)". Good result: a recent time, result 0. Bad result: a time older than your delete, a non-zero result, or no schedule from Get-DedupSchedule. More detail on job failures is in Event Viewer under \Applications and Services Logs\Windows\Deduplication\Operational.

Get-DedupJob lists running and queued jobs. Get-DedupVolume returns the volume's settings; Microsoft's own example reads them with Select *.

For the optimization side, "OptimizedFilesSavingsRate" and "SavingsRate" are the two fields Microsoft names, and "a downward-trending optimization rate ... might indicate that the Optimization jobs are not keeping up".

Also read the schedule in Task Scheduler under Microsoft\Windows\Deduplication, where Microsoft says the jobs live.

Example (illustrative): fs01 has a 4,000 GiB volume D: with 3,400 GiB used and 600 GiB free. On Monday an administrator deletes old project folders of 800 GiB logical size. Of that, 640 GiB is chunk data that no other file uses; the other 160 GiB is shared with files that remain. Free space on Monday afternoon is still about 600 GiB, because the chunk store has not changed. Saturday's regular job frees 615 GiB, so 3,400 − 615 = 2,785 GiB is used and 4,000 − 2,785 = 1,215 GiB is free. A later full job frees the last 25 GiB, which is 25 ÷ 640 = 3.9% of the unreferenced data, inside Microsoft's "up to 5 percent more". After it, 3,400 − 640 = 2,760 GiB is used and 1,240 GiB is free.

Fix

Cause 1: run Garbage Collection now

Start-DedupJob -Volume "D:" -Type GarbageCollection -Priority Low -StopWhenSystemBusy

Microsoft recommends -Priority Low together with -StopWhenSystemBusy. Add -Wait to watch progress. The job can queue "if the server is running another job on the same volume".

Disruptive step: Garbage Collection reads and rewrites chunk-store containers, so expect disk load while it runs. Microsoft says full Garbage Collection "could adversely affect the volume's lifetime shadow copies and the size of incremental backup", and that high-churn or I/O-intensive workloads "may see a degradation in performance". To stop it, run Stop-DedupJob -Volume "D:", which cancels the volume's jobs. Microsoft does not say what happens to work already done, and space already reclaimed cannot be returned to the deleted files.

Cause 2: schedule missing

Find the schedule's name with Get-DedupSchedule. If it exists but is switched off, Set-DedupSchedule -Name "<name>" -Enabled $true re-enables it. If none exists, Microsoft's example creates one:

New-DedupSchedule -Name "WeeklyGarbageCollection" -Type GarbageCollection -DurationHours 23 -Memory 100 -Cores 100 -Priority High -Days @(6) -Start (Get-Date "2016-08-13 07:00:00")

That example is written for a host that is idle on weekends. Change -DurationHours, -Memory, -Cores, -Priority, -Days and the start time to suit your server's quiet window. Microsoft notes that "the date part of the System.Datetime provided to -Start is irrelevant".

Risk: a schedule using all cores and memory at high priority for up to 23 hours can slow other work. Start lower. Back out with Remove-DedupSchedule -Name "WeeklyGarbageCollection", or Set-DedupSchedule -Name "<name>" -Enabled $false.

Cause 3: a failed job

Act on LastGarbageCollectionResultMessage and the Event Viewer log, then start the job again as above. Do not repeat it blindly: a job that failed once for lack of resources will fail again at the same time of day.

Causes 4, 5 and a leftover after a crash: run a full pass

Start-DedupJob -Volume "D:" -Type GarbageCollection -Full

Microsoft says full Garbage Collection "will recover 100 percent of the available space that can be recovered" and that it "may be used if full Garbage Collection is desired on a Backup volume". Its cmdlet page recommends running regular jobs routinely "and then once a month" running the job with -Full.

Disruptive step: a full pass compacts every container with any unreferenced chunk, so it costs more time and system resources than a regular pass and can enlarge the next incremental backup. If your backup depends on small incrementals, run it just before a full backup, not before an incremental. Stop it with Stop-DedupJob; there is no sourced way to undo work already done.

If full passes were disabled on purpose, ask who decided that and why before running one.

Cause 6: a queued or running job

Wait, or cancel your own queued job with Stop-DedupJob.

Cause 7: optimization behind

Garbage Collection will not help here. Check LastOptimizationResult and LastOptimizationTime. For volumes full of newly written data, see the capacity article on mapping Windows volumes to virtual disks and datastores.

Confirm the fix

  1. Get-DedupStatus shows a new LastGarbageCollectionTime and LastGarbageCollectionResult of 0.
  2. The free space reported for the volume rises by roughly the size of the data nothing else referenced. If it rises by far less than the deleted data's unique portion, and a full pass has finished, the deleted files were largely shared with files you kept.
  3. The next backup completes and is not unexpectedly large.

What to watch afterwards

  • Alert on a non-zero LastGarbageCollectionResult, and on a LastGarbageCollectionTime older than two scheduled weeks.
  • Run a full pass monthly if Microsoft's recommendation suits your backup size, or document why it is disabled.
  • After any large delete, expect the space back only after the next job. Plan capacity on that basis.

Risks and limits

  • These commands have not been run or lab-tested. They come from Microsoft Learn pages that apply to Windows Server 2016 to 2025.
  • The cmdlet pages do not list the properties of the status and volume objects. Only the field names Microsoft names in prose are used here; read the rest from your own Format-List output.
  • The percentage that decides when a regular job compacts a container is not published on these pages, so the example uses made-up figures.
  • Space held outside the chunk store is not covered here. Free space on the volume is also affected by things Microsoft does not describe on these pages.
  • Full Garbage Collection and its effect on shadow copies and incremental backups are described only in general terms by Microsoft.

Sources