Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Nightly backup suddenly much bigger: find what changed before blaming the backup

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

An incremental backup is a measurement of how much of the disk was written since the previous run. When it jumps from a few gigabytes to a few hundred, something wrote to the disk, and the backup is only reporting it. Check for ransomware first, because encryption rewrites files at scale and looks like this. If that is ruled out, compare how many files changed with how much the backup grew: the gap tells you whether data was written or whether blocks were only moved.

The checks below read only. They tell you which cause to chase, and each cause has a more detailed fix elsewhere.

What the symptom means

Block-level backups track changes by position on the disk. Arcserve's documentation says its block-level backups continue to "incrementally back up all changed blocks", including blocks that moved in a defragmentation "even if no data has changed in the files". Acronis says an incremental or differential made after a defragmentation, or after a restore, is considerably larger than usual, because each changes file locations on the disk. A backup that reads files by name and modified time sees things differently. Neither vendor's page covers that case, so its behaviour is not stated here; check how your product reads changes.

That gives a useful split:

  • Few files changed, backup large. Blocks were moved or rewritten without a matching change to file contents: defragmentation, encryption of the volume, deduplication, a restore, or the backup losing track of what it had already stored.
  • Many files changed, backup large. Data was genuinely written: ransomware, a bulk import or migration, big logs, or a database rewriting itself.

Look for ransomware first

CISA's #StopRansomware Guide (September 2023) says many ransomware variants "attempt to find and subsequently delete or encrypt accessible backups", and tells responders to look for endpoint changes that impair recovery: "anomalous usage of built-in Windows tools such as bcdedit.exe, fsutil.exe (deletejournal), vssadmin.exe, wbadmin.exe, and wmic.exe (shadowcopy or shadowstorage)". It also refers to ransom notes and encrypted files. A large nightly backup alongside unfamiliar file extensions, ransom notes, or shadow copies that have vanished is a different problem from the other causes below.

If any of that is present, stop here. Do not run clean-up, pruning or restores. Follow the recovery steps in the related article on recovering after ransomware, and do not treat the newest backups as clean.

Likely causes

Cause What you see How to confirm
Ransomware or mass encryption Very many files with recent modified times; unfamiliar extensions or notes; shadow copies gone Listing by last write time (below); the tool-misuse signs CISA lists
Defragmentation or Optimize Drives Large increment, few modified files, the day after a maintenance run Optimize Drives "last run" time; see the related article on defrag and backup size
BitLocker turned on, or full-volume encryption Large increment on the day encryption ran, few modified files manage-bde -status conversion status and percentage encrypted
Windows Data Deduplication enabled or newly optimising Large increment on a file server volume after the feature was enabled Get-DedupStatus, Get-DedupJob
SQL Server index rebuild Database files rewritten; big increments after the maintenance job Maintenance plan or job schedule against the backup dates
Genuine new data (import, migration, logs, exports) Many files with recent modified times and sizes that add up to the increment Sum of modified file sizes (below)
Restore onto the volume before the backup Increment larger after a restore Restore history and date
Change tracking lost, so the product re-read the disk Full-sized read, even if few blocks changed Job log; see the related article on change tracking resets

Prove the cause before changing anything

All of these only read. They can be heavy on a large volume, so run them out of hours.

  1. Get the numbers from the backup. For the last two weeks, note per run the date, whether the product reported it as an incremental or a full, the amount read and the amount stored. A jump on one date points at one event. A steady climb points at growth.

  2. Count and sum the files written recently. From PowerShell on the server:

Get-ChildItem -LiteralPath D:\ -Recurse -File -Force | Where-Object { $_.LastWriteTime -gt (Get-Date).AddDays(-1) } | Measure-Object -Property Length -Sum

Get-ChildItem with -Recurse and -File lists files, Where-Object filters them, and Measure-Object -Property Length -Sum totals their sizes. Microsoft's Get-WinEvent page uses (Get-Date).AddDays(-2) for a date two days back; AddDays(-1) here is one day. The result shows a count and a sum of bytes. Change the window to match the gap between two backups.

  1. List the largest recent files.
Get-ChildItem -LiteralPath D:\ -Recurse -File -Force | Where-Object { $_.LastWriteTime -gt (Get-Date).AddDays(-1) } | Sort-Object -Property Length -Descending | Select-Object -First 20 -Property FullName, Length, LastWriteTime
  1. Compare the sum with the increment. If the files written in the window add up to roughly the size of the increment, data was written. If the sum is a small fraction of the increment, blocks were moved or rewritten, or the product re-read the disk.

  2. Read the shape of the list. Thousands of files across many folders all changed within a short time, with unfamiliar extensions or the same new file in many folders, matches the signs of ransomware and means you should go back to the section above. A few very large files (a database, a virtual disk, a mailbox store, an export) is the usual picture of real growth.

  3. See what the system was told to change. NTFS keeps a change journal. Microsoft says the USN change journal "provides a persistent log of all changes made to files on the volume". fsutil usn queryjournal D: reports its state and capacity, and fsutil usn readjournal D: reads its records. The output is large, so send it to a file. Microsoft's page does not show the record layout, so read a small sample on your own system first.

    Do not run fsutil usn deletejournal: CISA lists it among the tools ransomware uses to impair recovery, and Microsoft warns that deleting the journal "impacts the File Replication Service (FRS) and the Indexing Service".

  4. Check the maintenance candidates.

    • Optimize Drives: the "last run" time per volume (Optimize Drives app), and defrag D: /a /v for an analysis that does not change the volume.
    • BitLocker: manage-bde -status reports "conversion status" and "percentage encrypted" for all drives.
    • Deduplication: Get-DedupStatus returns status for volumes with deduplication metadata, and Get-DedupJob returns running or queued jobs.
    • SQL Server: check the schedules of index maintenance jobs against the backup dates.

Example (illustrative): fs01 has 400 GB used on D:. Nightly incrementals run about 6 GB (1.5% of 400 GB). On Wednesday the increment is 96 GB, 16 times normal (96 / 6 = 16). The listing shows about 3,000 files written in the last day, totalling 5 GB. Five gigabytes of changed files would not normally produce 96 GB of changed blocks, so something moved blocks. Optimize Drives shows a run at 03:00 on Tuesday, after the 01:00 backup, so Wednesday's backup is the first to see it. The cause is a defragmentation, not data growth and not the backup product, and the next increment is back near 6 GB.

Fix

Ransomware or mass encryption

Isolate the affected servers from the network and follow the recovery steps in the related article. Risk: isolating or powering off servers changes what can be preserved for investigation, and stopping a backup job mid-run may leave an incomplete restore point. Decide with whoever is running the response. There is no safe automatic step here.

Defragmentation or Optimize Drives

Move the task away from the backup window, and do not disable it blindly. The related article on defrag and backup size gives the details, including the effect on shadow copies.

Disruptive step: moving the Optimize Drives schedule changes when volumes are optimised. Impact: the defragmentation runs at the new time, and the first backup after it is the larger one, so the new time must not sit just before the backup. To back out, set the schedule back to its old time; note the old setting before you change it.

BitLocker or full-volume encryption

Microsoft's manage-bde -on page says that without -usedspaceonly, "all used space and free space on the volume will be encrypted", while Used Space Only encrypts "the sections of the volume containing used space" and not free space. A block-level backup sees the encrypted blocks as written. That follows from the block tracking described above; Microsoft's page does not state it. Plan the one-off increase: it is roughly the volume's encrypted blocks (the used space with -usedspaceonly, the whole volume without it), and it does not repeat once the conversion has finished. Take a new full backup afterwards if your product recommends one.

Data Deduplication

Microsoft says Data Deduplication optimises data after it is written, places chunks in the chunk store and replaces "the original file stream of now optimized files with a reparse point". The first optimisation pass after enabling it works through existing data, and how your backup product sees that is product-specific; Microsoft's page does not say. Treat it as a one-off increase and enable deduplication outside the backup window. Risk: enabling or disabling deduplication on a production volume changes how its files are stored, so make sure a tested restore exists first.

SQL Server index rebuild

Microsoft says "rebuilding an index drops and re-creates the index", and recommends reorganising where there is no specific reason to rebuild, because it is "less resource intensive". How much a rebuild adds to a backup depends on your databases and your product; Microsoft's page does not quantify it. Space the maintenance away from backups and rebuild only the indexes that need it.

Disruptive step: changing the index maintenance schedule or scope changes when and which indexes are rebuilt. Impact: indexes you stop rebuilding can stay fragmented, which may slow queries. To back out, restore the previous job schedule and scope; note them before you change anything.

Genuine new data

Decide whether it is meant to be backed up. If a temporary export or scratch folder is the cause, move it to a volume the backup does not cover, or exclude it in the backup job. Risk: excluding a path means it is no longer protected. Do this only for data you can recreate.

A restore, or lost change tracking

After a restore Acronis advises a new full backup before further incrementals. For a reset of change tracking, follow the related article on that fault.

Confirm the fix

  1. The next increment is back in the normal range for the volume.
  2. The files-written sum from step 2 and the increment now agree to within the usual overhead.
  3. No new ransom notes or unfamiliar extensions, and shadow copies are as expected.
  4. A test restore of one recent file from the large backup matches the source.

What to watch afterwards

  • Increment size per run, with a flag at several times the recent average.
  • A calendar of maintenance windows (Optimize Drives, encryption, deduplication, index jobs) beside the backup schedule.
  • Large recent files that you did not expect.
  • Shadow copy count and the tools listed by CISA in the event log.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from Microsoft Learn, CISA and vendor documentation.
  • The file listing in step 2 cannot detect a block-level change that leaves a file's modified time alone; that is why the comparison with the increment matters.
  • Arcserve and Acronis describe their own block-level products. Other products read changes differently, and the size of the effect varies.
  • The CISA guide is dated September 2023 and shows no revision date; it lists tool misuse as a sign, not a test, and the same tools are used legitimately by administrators.
  • The Data Deduplication page is dated February 2022; check current behaviour on your Windows Server version.
  • The sizes in the example are invented to show the arithmetic.

NetLeverage includes a daily full disaster-recovery backup of each hosted server at no extra charge. Talk to an engineer.

Sources