Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

S3 backup bucket keeps growing after backups are deleted: noncurrent versions and delete markers

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

If versioning is switched on for the bucket, deleting a backup file does not remove it. S3 adds a delete marker, the file becomes a noncurrent version, and you keep paying for its full size. A backup tool that "deletes expired backups" every night therefore frees nothing. The bucket only shrinks when a lifecycle rule permanently expires noncurrent versions.

The delete marker itself is almost free. The cost is the hidden copies underneath it. Measure them first, then choose how long you want deleted data to stay recoverable.

What the symptom means

AWS describes versioning as keeping multiple variants of an object in one bucket. In a versioning-enabled bucket, "if you delete an object, Amazon S3 inserts a delete marker instead of removing the object permanently. The delete marker becomes the current object version." An overwrite creates a new version instead of replacing the old one.

Two sentences from the S3 User Guide explain the bill. First: "Each version of an object is the entire object; it is not just a diff from the previous version. Thus, if you have three versions of an object stored, you are charged for three objects." Second, on delete markers: "A delete marker doesn't have data associated with it", and the storage charged for one "is equal to the size of the key name".

Two things follow:

  • Ordinary listings hide the problem. A ListObjectsV2 request does not return objects whose current version is a delete marker, so a file browser or the backup tool's own inventory looks empty or small while the bucket total stays high. Only a ListObjectVersions request (the console's Show versions switch) shows markers and old versions.
  • Once a bucket has been version-enabled it can never return to unversioned. You can only suspend versioning, which stops new versions accruing and leaves existing ones in place.

A rule that used to work can also stop working. The guide warns that if an unversioned bucket had an object-expiration rule and you later enable versioning, you "must add a noncurrent expiration configuration" to keep the same permanent-delete behaviour. An Expiration action on a versioned bucket only adds a delete marker.

Likely causes

Cause What you see How to confirm
Backup tool deletes expired files with a plain delete Bucket total keeps rising while the tool reports a steady number of files list-object-versions shows DeleteMarkers entries and Versions entries with IsLatest false
Tool rewrites the same keys (index or manifest files) Many versions of a few keys Repeated Key values in Versions, each with a different VersionId
Lifecycle has only Expiration, no NoncurrentVersionExpiration Rule exists and "works" on current files, but total never falls Read the lifecycle configuration and look for the noncurrent action
Noncurrent rule exists but is set very long, or keeps many versions Space returns slowly or only partly NoncurrentDays and NewerNoncurrentVersions values; both must be exceeded before deletion
Rule disabled, or its prefix, tag or size filter does not match Nothing expires Status is Disabled, or the Filter excludes the keys the tool writes
Locked versions Rule is correct but some versions stay Object Lock retention or legal hold on those versions
Leftover incomplete multipart uploads Bytes are counted but no object is visible See the large-upload article; they need their own abort rule

On locked versions, AWS states that lifecycle rules "continue to function normally on protected objects, including placing delete markers. However, a locked version of an object cannot be deleted by a S3 Lifecycle expiration policy."

Prove the cause before changing anything

These commands read only. They are taken from the AWS CLI reference and have not been run.

  1. Measure, do not count. In the S3 console, S3 Storage Lens includes the free-tier metric NonCurrentVersionStorageBytes ("Number of noncurrent version bytes") and NonCurrentVersionObjectCount, plus DeleteMarkerObjectCount. CloudWatch publishes BucketSizeBytes once a day and states that it covers "both current and noncurrent objects". Compare noncurrent bytes with total bytes. If noncurrent bytes are most of the total, you have found the growth.

  2. Sample the versions for one prefix. The call is paginated, so ask for a small page:

aws s3api list-object-versions --bucket backup01-bucket --prefix daily/ --max-items 50

Good result: entries under Versions with IsLatest true for files you expect, and few others. Bad result: many Versions entries with IsLatest false, or DeleteMarkers entries with IsLatest true, for files the backup tool believes it has removed. The reference lists Size for each version, so you can see how large the hidden ones are.

  1. Read the lifecycle configuration and save it to a file before you do anything else:
aws s3api get-bucket-lifecycle-configuration --bucket backup01-bucket

Look for NoncurrentVersionExpiration, the Status of each rule, and the Filter.

Example (illustrative): backup01 stores nightly files in a versioned bucket. The tool keeps 30 days and removes the expired 25 GB each night, so the live set is 30 x 25 = 750 GB. After a year the tool has uploaded 365 x 25 = 9,125 GB and removed 8,375 GB of it by plain delete. Every removed file is still a noncurrent version, so the bucket holds all 9,125 GB, about 12 times what the tool reports. With noncurrent versions expiring 7 days after they become noncurrent, steady state is about 750 + 7 x 25 = 925 GB (a little more, because expiry times round up to the next midnight UTC), roughly a tenth of the 9,125 GB.

Fix

Cause: no rule for noncurrent versions

Choose the recovery window first. The window is how long you can still undo a delete or an overwrite, whether by mistake or by an attacker. AWS gives the same reasoning: "Delayed removal of noncurrent objects can be helpful when you need to correct any accidental deletes or overwrites."

The lifecycle JSON below uses fields shown in the put-bucket-lifecycle-configuration reference. The 7 days is an illustration, not a recommendation.

{
  "Rules": [
    {
      "ID": "expire-noncurrent-and-markers",
      "Filter": { "Prefix": "" },
      "Status": "Enabled",
      "NoncurrentVersionExpiration": { "NoncurrentDays": 7 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}

NoncurrentDays counts from the moment a newer version is created (the successor), rounded up to the next midnight UTC. ExpiredObjectDeleteMarker removes a marker once no noncurrent versions remain beneath it. The lifecycle documentation puts both actions in one rule in its example, and says the marker action cannot be used with a tag-based filter.

Disruptive step: applying the rule. put-bucket-lifecycle-configuration "will overwrite an existing lifecycle configuration", so merge your saved rules into the new file first. AWS also says a new configuration applies "to both existing objects and objects that you add", so the first run can permanently delete years of accumulated noncurrent versions. AWS describes the result of NoncurrentVersionExpiration as: "These deleted objects can't be recovered." There is no way back. To limit the blast radius, start with a narrow prefix, or keep a few versions with NewerNoncurrentVersions (1 to 100; the rule then needs a Filter element).

aws s3api put-bucket-lifecycle-configuration --bucket backup01-bucket --lifecycle-configuration file://lifecycle.json

Cause: tool rewrites the same keys

The same rule applies. If you need only the newest copy of those keys, a short window is enough.

Cause: long window or high version count

Both NoncurrentDays and NewerNoncurrentVersions must be exceeded before a version is deleted. Lower one, or both, and apply the same cautions.

Cause: disabled or mismatched rule

AWS: "If a rule is disabled, Amazon S3 doesn't perform any actions specified in the rule." Correct the Status or the Filter. A rule with a prefix filter will not touch keys outside the prefix.

Cause: locked versions

Wait for the retention to end, or change the design. The linked S3 Object Lock article explains what each mode allows. Lifecycle cannot remove a locked version.

Cause: incomplete multipart uploads

They need their own AbortIncompleteMultipartUpload rule, because AWS says "Object expiration lifecycle configurations don't remove incomplete multipart uploads." The large-upload article covers it.

Confirm the fix

  1. Allow time. AWS notes "there may be a delay between the expiration date and the date at which Amazon S3 removes an object", and that "you are not charged for expiration or the storage time associated with an object that has expired".
  2. Watch NonCurrentVersionStorageBytes and the daily BucketSizeBytes fall over the following days.
  3. Re-run the version sample for a prefix. Old IsLatest false entries older than your window should be gone.
  4. Do a trial restore of a file the backup tool still holds, to prove the rule did not remove anything the tool needs.

What to watch afterwards

  • Alert on noncurrent bytes as a share of total bytes, and on DeleteMarkerObjectCount.
  • Keep the lifecycle JSON in version control. It is the only record of your recovery window.
  • Limit who may change lifecycle rules. AWS says a bucket policy cannot stop a lifecycle rule from deleting: "even if your bucket policy denies all actions for all principals, your S3 Lifecycle configuration still functions as normal". The put-bucket-lifecycle-configuration reference lists s3:PutLifecycleConfiguration among the permissions to deny if you want to stop users removing objects. Someone who can write a short rule can erase the recovery window.
  • Re-check after any change to the backup tool's retention. A tool that deletes more often creates more versions.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from the AWS documentation.
  • Permanent expiry is irreversible. A shorter window saves money and shortens the time in which you can undo a delete.
  • Editing a lifecycle configuration created before September 2024 can change its default for small-object transitions: AWS states that if you "create, edit, or delete rules, the default transition behavior for your configuration changes to the updated behavior", which stops objects under 128 KB moving to another storage class. Check any transition rules you also run.
  • This page covers an unlocked, versioned bucket. A locked bucket behaves differently, as described in the linked Object Lock article.
  • The AWS pages show no dates, and AWS can change lifecycle behaviour. Compare the put-bucket-lifecycle-configuration reference for your CLI version.
  • Other S3-compatible services may implement versioning and lifecycle differently from AWS.

Sources