Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Locked backup storage that only grows: find the retention or legal hold that is keeping the data

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

When backup storage is locked, the bill stops following your retention policy and starts following the lock. A version stays stored, and stays billed, until its own retain-until date has passed and every legal hold on it has been cleared. The backup software's clean-up then either fails with an access-denied error or "succeeds" by adding a delete marker that frees nothing.

The way out starts with finding which lock holds the data: a bucket default that is too long, a retention date the backup product set itself, a legal hold nobody cleared, or an event hold that is still on. What you may change depends on the mode, and in compliance mode the answer for existing data is to wait.

What the symptom means

In Amazon S3, Object Lock works only in buckets with versioning enabled, and the lock sits on each object version, not on the object name. A retention period or legal hold "protects only the version that's specified in the request." It does not stop new versions or delete markers being added on top.

That explains the two faces of the fault:

  • A permanent delete, one that names a version ID, returns Access Denied (403 Forbidden) while the version is locked.
  • A simple delete, one that names no version, returns 200 OK and inserts a delete marker. The data is still there, and Amazon documents that delete markers are not WORM-protected.

Backup clean-up tools usually issue the second kind. The job reports success, the catalogue says the old backup is gone, and the storage figure does not move.

Storage is charged on every version. Amazon states that normal S3 rates apply for every version stored, including noncurrent ones, and that each version is a whole object, not a difference from the last. A lifecycle rule cannot rescue this: the lifecycle documentation says S3 "doesn't take any action on noncurrent versions of objects that have the S3 Object Lock configuration applied."

What each lock lets you change

Lock Can the date be shortened? Who can remove it Notes
Compliance retention No. Mode and period cannot be shortened or changed Nobody, including the account root user. Amazon states the only way to delete before the date is to delete the AWS account Anyone with s3:PutObjectRetention can extend it
Governance retention Yes, with s3:BypassGovernanceRetention and the bypass header Users holding that permission The S3 console sends the bypass header by default if you hold the permission
Legal hold Not applicable; it has no end date Anyone with s3:PutObjectLegalHold Independent of retention. A hold outlives an expired retention period
Event hold (variable retention) The date cannot move earlier Releasing the hold fixes the date at release time plus the configured duration While the hold is on, the retain-until date keeps moving forward

The other two large providers behave similarly but not identically. Microsoft documents that a locked time-based retention policy can be extended but not decreased, and cannot be deleted; an unlocked policy can be shortened or deleted. Google documents that once a bucket's retention policy is locked "you cannot remove it or reduce the retention period it has", though you can increase it. Both providers state that an unlocked policy can still be changed, which is why a policy should be locked only after its length has been proved right.

Likely causes

Cause What you see How to confirm
Bucket default retention longer than the backup retention Every new version carries a long retain-until date, set when it was written get-object-lock-configuration shows the default mode and days or years
The backup product set its own retention on each object Dates differ from the bucket default. An explicit per-object setting overrides the bucket default head-object on a few versions shows each date
Legal hold left on Retain-until has passed but the version still cannot be deleted get-object-legal-hold returns ON
Event hold still on Retain-until date moves forward every time you read it get-object-retention shows EventHold as ON
Retention extended repeatedly Old versions' dates move later after each maintenance run Compare the same version's date over two days
Deletes only add delete markers Clean-up reports success, space does not fall list-object-versions shows a long DeleteMarkers list above many Versions
Governance mode and nobody holds the bypass permission 403 on permanent delete, though dates are not far off The mode shows GOVERNANCE, and the clean-up account's policy lacks s3:BypassGovernanceRetention

Prove the cause before changing anything

All of these commands only read. They appear in the AWS CLI reference; output layouts below are described, not captured.

  1. The bucket default. This needs s3:GetBucketObjectLockConfiguration.
aws s3api get-object-lock-configuration --bucket backup01-archive

A good result shows a mode and a period that match your written backup retention. A bad one shows a period far longer, such as years where days were intended. The output nests the settings under ObjectLockConfiguration, Rule and DefaultRetention.

  1. A sample of versions, oldest and newest.
aws s3api list-object-versions --bucket backup01-archive --prefix <prefix>
aws s3api head-object --bucket backup01-archive --key <key> --version-id <version-id>

list-object-versions returns Versions and DeleteMarkers. head-object returns the lock mode, the retain-until date and the legal hold status, but only if the caller holds s3:GetObjectRetention and s3:GetObjectLegalHold. A missing field can mean missing permission, not missing lock.

  1. A legal hold or event hold on a specific version.
aws s3api get-object-legal-hold --bucket backup01-archive --key <key> --version-id <version-id>
aws s3api get-object-retention --bucket backup01-archive --key <key> --version-id <version-id>

The first returns LegalHold with a Status of ON or OFF. The second returns Retention with Mode, RetainUntilDate and, where variable retention is in use, EventHold and EventHoldDuration.

  1. For a bucket-wide picture, use S3 Inventory. It can include retain-until date, retention mode, legal hold status, event hold status and duration for every object. Reports can take up to 48 hours to generate, so a retain-until date for an active event hold reflects report time, not today.

On Google Cloud Storage, gcloud storage buckets describe gs://BUCKET_NAME --format="default(retention_policy)" shows the bucket's retention policy.

Example (illustrative): a backup tool writes 100 GB of new data each day to bucket backup01-archive and is meant to keep 35 days. Someone sets the bucket default to compliance mode, 365 days. At steady state the intended store is 100 GB × 35 = 3,500 GB. The locked store would be 100 GB × 365 = 36,500 GB, about 10.4 times larger, or 33,000 GB over. The fault is noticed on day 60, with 6,000 GB written. Changing the default to 35 days affects what is written from then on, because the retain-until date is worked out when each version is written. The 6,000 GB stays locked: the data written on day 1 until about day 366, the data written on day 60 until about day 425. The store peaks near 6,000 + 3,500 = 9,500 GB around day 95 and stays there until day 366, then falls by about 100 GB a day. Each month the extra 6,000 GB sits there it costs 6,000 × R, where R is the provider's per GB-month rate.

Fix

Work in this order: stop adding to the problem, then release what the mode allows, then wait out the rest.

Stop new data being over-locked

If the bucket default is wrong, set the right one. Amazon documents this command shape:

aws s3api put-object-lock-configuration --bucket backup01-archive --object-lock-configuration='{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 35 }}}'

Risk: this writes the mode as well as the period. Choosing compliance mode here means every new version is locked in a mode that nobody can shorten. Choose governance while you prove the period right, if your threat model allows it. Backing out: you can modify the default again, but versions already written keep the retention they were given.

If the backup product sets retention on each object, its own setting overrides the bucket default. Correct the retention setting inside the product, not only the bucket.

Release a legal hold

aws s3api put-object-legal-hold --bucket backup01-archive --key <key> --version-id <version-id> --legal-hold="Status=OFF"

Disruptive step: a legal hold may exist because someone with authority placed it for a reason. Confirm with whoever placed it before clearing it. Once cleared, the version is protected only by any remaining retention date. There is no way back for data that is later deleted.

Release an event hold

put-object-retention with {"Mode":"COMPLIANCE","EventHold":"OFF"} releases an event hold. Risk: releasing does not free the data. Amazon sets the final retain-until date at release time plus the configured duration, so the object stays protected for the full duration afterwards.

Governance mode data

A user with s3:BypassGovernanceRetention can shorten the date, or delete the version, by adding the bypass option:

aws s3api delete-object --bucket backup01-archive --key <key> --version-id <version-id> --bypass-governance-retention

Disruptive step: this permanently deletes the version. The S3 documentation says a permanently deleted version cannot be recovered. Do it only for versions you have listed and checked against the backup catalogue, and never with a broad credential that backup software or an attacker could reuse. Bypassing governance mode does not clear a legal hold.

Compliance mode data

For versions already locked in compliance mode there is no change to make. The retain-until date cannot be shortened, and nobody can delete the version early. Amazon documents deleting the AWS account as the only route, which is not a clean-up option for a live backup account. Budget for the extra storage, as in the example, and make sure the bucket default no longer adds to it.

Delete markers and noncurrent versions

Once locks have expired, a lifecycle rule using NoncurrentVersionExpiration is how S3 permanently removes noncurrent versions, and an expired object delete marker can be removed by an Expiration action. Amazon notes a deleted version "can't be recovered", so test any rule on a prefix holding disposable data first.

Confirm the fix

  1. get-object-lock-configuration shows the intended default.
  2. A version written after the change shows a retain-until date about the intended number of days ahead of its write date.
  3. get-object-legal-hold returns OFF for versions you cleared.
  4. After the oldest wrongly locked versions expire, the storage metric falls. Compare a count of noncurrent versions before and after, using S3 Storage Lens or an Inventory report.

What to watch afterwards

  • Alert on retention changes. Amazon documents an s3:ObjectRetention:Put event notification for explicit retention calls, and S3 Event Notifications with CloudTrail as the way to track access and changes to Object Lock configurations.
  • Keep a written number: the lock period you intend, next to the backup retention you promised. They should differ only by the safety margin you chose on purpose.
  • Track the noncurrent-version byte count monthly. A flat line means clean-up works.
  • Restrict s3:PutObjectRetention and s3:PutObjectLegalHold to a small group. Amazon documents that any user with permission to place a retention period can extend one, and an extension adds storage cost.
  • Use a bucket policy limit if you want a ceiling. Amazon documents s3:object-lock-remaining-retention-days as a condition key for capping retention lengths on PutObjectRetention.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from the vendors' documentation.
  • The Amazon pages show no revision dates. Object Lock options such as variable retention are recent, so an older SDK or backup product may not understand them.
  • A bucket default is applied when a version is written. The documentation does not say that changing it rewrites existing versions; confirm on a sample version.
  • Compliance-mode retention cannot be shortened by anyone. Treat the mode as a decision about cost as well as protection.
  • Azure and Google behave differently in detail. An unlocked policy can still be changed on both, and a locked one cannot be shortened on either. Check the vendor pages before relying on a table here.
  • Backup products differ in how they set and extend locks. The vendor's own documentation, not a general rule, says what a given product does.

Sources