Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Backup storage bill keeps growing although the data has not: find the cause

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When backup storage grows while the server it protects stays the same size, the extra space is almost always data the backup system is still holding, not data the server created. Seven things cause it: a retention period longer than intended, more daily change than before, a delete step that removes the index entry but not the data, versioned storage that keeps every old copy, a lock that has not expired, a retention rule that never matches some old backups, and backups kept after protection was switched off.

Do not shorten retention or delete anything first. Read the stored figures, work out which cause fits, and only then change one setting.

What the symptom means

A backup store grows by the data that is new to it. Every backup point also keeps alive any older block it still refers to. Space comes back only when the last point that refers to a block is removed and a clean-up step then deletes the block itself.

Two figures are easy to confuse. The protected size is how much data the source holds. The stored size is what the backup store actually occupies. restic's own manual describes the difference: its stats command has a restore-size mode (the size of the restored files) and a raw-data mode that "counts the size of the blobs in the repository, regardless of how many files reference them". Comparing the two "can tell you how much deduplication is helping you".

A growing bill with a flat source therefore means the stored figure is rising faster than the protected one. The question is why, and the causes below separate cleanly.

Likely causes

Cause What you see How to confirm
Retention is longer than intended, or the rule was never applied Oldest restore point is older than the written policy; point count keeps rising List the points and compare with the policy
Daily change has risen Each new point adds more than it used to Compare the new data added by recent runs against runs from months ago
Delete runs but clean-up does not Points disappear from the list, stored size stays flat or rises Compare stored size before and after a delete; use the tool's dry-run on clean-up
Versioned object storage Deleting from the backup tool adds markers; old versions remain billed in full Count versions and noncurrent bytes in the bucket
A lock has not expired Delete attempts are refused, or stored size cannot fall Read the retention date and any legal hold on the objects
Retention rule never matches some points A retired server or renamed folder keeps its last points for ever List points by host and path; compare with the policy grouping
Protection stopped, data kept The bill continues for a server nobody backs up now Check the vault or repository for data belonging to stopped sources

Prove the cause before changing anything

Start with the commands that only read. These commands are taken from vendor documentation; the later --dry-run options print what would happen and change nothing.

restic (documentation for 0.19.1):

restic snapshots
restic stats --mode raw-data latest
restic stats latest

The first lists every point with its date, host and path. The second counts stored blobs; the third (default restore-size) counts the size needed to restore. restic's manual says to run the same stats commands per host with --host if several servers share the repository. restic locks the repository for some operations: the documentation says that during a prune "the repository is locked and backups cannot be completed", so run the checks outside the backup window.

To see what a policy would remove without removing it, forget accepts --dry-run, which "instructs restic to not remove anything but instead just print what actions would be performed".

Borg (documentation for 1.4.5):

borg info REPOSITORY::ARCHIVE
borg compact --dry-run REPOSITORY

For borg info, "this archive / deduplicated size" is the data stored only for that archive, and "all archives / deduplicated size" is the data in the whole repository. The two do not add up, by design. borg compact has a -n, --dry-run option.

Amazon S3 and compatible stores with versioning:

aws s3api list-object-versions --bucket BUCKETNAME --prefix PREFIX

The AWS CLI reference describes this as returning "metadata about all versions of the objects in a bucket". The result lists current and noncurrent Versions and DeleteMarkers. For a bucket-wide figure, S3 Storage Lens has "Noncurrent version bytes" and "% noncurrent version bytes" metrics.

Windows Data Deduplication: Microsoft describes a Garbage Collection job that "reclaims disk space by removing unnecessary chunks that are no longer being referenced by files that have been recently modified or deleted". If files were deleted and the volume did not free space, that job has either not run or not finished.

Example (illustrative): fs01 holds 800 GB. About 16 GB of that is new or changed each day, and every changed block is new to the store. The policy is "keep 30 daily points". A first backup stores 800 GB and is the oldest of the 30 points, so the expected stored size is 800 GB + (29 x 16 GB) = 1,264 GB. After 80 nightly backups the repository measures 2,064 GB, which is 800 GB + (79 x 16 GB). The script runs forget nightly but never prune, so the daily changes from the 50 days that have aged out are still stored. The unused space is 50 x 16 GB = 800 GB. At a rate of R per GB-month, that is 800 x R every month for data nobody can restore. The numbers are made up, but the sum is the check: expected stored size against measured.

Fix

Change one cause at a time, and re-measure before moving to the next.

Retention longer than intended

Compare the written policy with what the tool is applying. For restic, the manual says the policy options are combined so "a snapshot only needs to match a single option to be kept". An extra --keep-yearly or --keep-within-* option therefore keeps points a short --keep-daily alone would not. Remove the option you did not mean, and run forget --dry-run first.

Disruptive step: shortening retention deletes older restore points and cannot be undone once clean-up runs. The only way back is to restore from a copy elsewhere. Confirm the business needs, and any legal retention duty, before you shorten.

More daily change

If the source now changes more, the stored figure rises honestly. Find what is writing: new database exports, a large log directory, a folder that now holds compressed archives. Excluding a folder that need not be protected is a change to scope, so agree it with whoever owns the data.

Risk: an exclusion changes what the next backups contain. From that run on, the folder has no new restore points, so a file lost or damaged afterwards cannot be restored from them. To back out, remove the exclusion and run a backup; points made while it was excluded stay without that folder. Azure's own estimator page makes the same point: "Higher churn will mean that you back up more data".

Delete runs, clean-up does not

In restic, "two commands need to be called in sequence: forget to remove snapshots, and prune to remove the remaining data that was referenced only by the removed snapshots". forget --prune runs both. In Borg 1.4.5, "repository disk space is not freed until you run borg compact". Add the missing step to the schedule, in a window when no backup runs.

Disruptive step: prune and compact permanently remove data. Run the dry-run form first. restic also keeps some unused data on purpose: by default prune allows up to 5% of the repository to remain unused, to avoid rewriting packs.

Versioned object storage

S3 documents that "each version of an object is the entire object; it is not just a diff from the previous version", and that a delete in a versioned bucket inserts "a delete marker instead of removing the object permanently". A lifecycle rule with NoncurrentVersionExpiration deletes noncurrent versions, and the documentation adds "These deleted objects can't be recovered." You can keep a number of newer versions with NewerNoncurrentVersions (1 to 100) and remove expired delete markers with ExpiredObjectDeleteMarker.

Disruptive step: a noncurrent-version expiry rule destroys the old versions it matches. Keep the days long enough to cover how late you might notice a bad overwrite. There is no way back.

A lock that has not expired

With S3 Object Lock in compliance mode, a protected version cannot be deleted by any user including the root user, "its retention period can't be shortened", and the only early route is deleting the AWS account. A legal hold "has no expiration date" and stays "until you explicitly remove it". The cost of a lock set too long is paying to store the data until it ends. Check what was set before it is applied to more objects.

A rule that never matches

restic groups points by host and paths, and applies the policy to each group, which its manual calls "a safety feature". It also counts only days that "have one or more snapshots". Read together, a server that was retired last year still keeps its final run of points under a count-based rule, because no later days compete with them. Borg's prune considers every archive in the repository unless a prefix or glob limits it. Remove retired groups deliberately, with a named forget, after confirming nobody needs them.

Disruptive step: a named forget removes those restore points from the repository, and the next prune or compact removes the data they alone referenced. Impact: the retired server can no longer be restored from this repository. There is no way back except a copy held elsewhere, so run with --dry-run first and keep a copy until the retention duty has ended.

Protection stopped, data kept

Microsoft says of Azure VM backup that billing "continues as long as any backup data for the VM is stored in a vault", even when protection is stopped. Delete the data of retired sources only when its retention duty has ended.

Disruptive step: deleting a retired source's backup data removes its restore points from the vault or repository. Impact: that server can no longer be restored, and the billing for it stops. There is no way back except a copy held elsewhere.

Confirm the fix

  1. The stored size (raw-data, borg info all-archives size, or bucket bytes) stops rising over two full retention cycles.
  2. After clean-up, stored size falls by about the amount the dry run predicted.
  3. The newest points still restore. Restore a sample to a scratch location and compare it with the source before you trust the new policy.

What to watch afterwards

  • Record stored size, protected size and point count each month. A ratio that drifts upward needs an explanation.
  • Alert if the oldest point is older than the policy, or if clean-up has not completed in the past week.
  • When a server is retired, write down who removes its backups and when.
  • Re-run the check after any change to schedule, folders or storage class.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from vendor documentation.
  • Borg's compact rule is documented for 1.4.5; newer major versions may behave differently.
  • The seven causes are a diagnostic list, not a ranking. Some systems show two at once.
  • A product report may show only a total stored size, without the steps behind it. Ask the provider which of the figures above it can show.

Sources