Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

A backup account that can add but not delete: append-only in S3 policies, restic and Borg

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

An append-only backup account is a credential that can write new backup data and read it back, but is refused when it tries to delete or overwrite what is already stored. If ransomware takes over the machine being backed up, it holds only that credential, so it cannot erase the history through it.

That is a narrower promise than "immutable". The deletion still has to happen somewhere, because storage that never frees space grows without limit. Whoever holds the clean-up credential can delete everything. The protection is only as strong as the separation between the two. It also has a cost: space comes back only when clean-up runs, and clean-up runs on your schedule, not the backup's.

What does append-only actually stop?

Each tool defines it differently, and the differences matter.

Mechanism What is refused What still works Where the rule is enforced
restic with rest-server --append-only Deleting or overwriting stored backup files. The README says the mode "allows creation of new backups but prevents deletion and modification of existing backups" Writing new files, reading, and deleting lock files On the server that holds the repository
Borg repository with append_only 1, or borg serve --append-only Overwriting or deleting committed data in the segment files borg delete and borg prune still run. borg compact does nothing, so space is not freed In the repository config, or in the server-side command tied to an SSH key
S3 bucket policy and IAM policy with no delete actions Whatever deletes the policy leaves out. Writing over a key is a separate question Writes, reads, listing In the storage service, at the credential

For restic's rest-server, the source (v0.14.0) shows how narrow the rule is. When the flag is set, a request to delete an object of any type other than locks returns 403 Forbidden, and so does a request to delete the repository config. Lock files can still be removed, an exemption that fits restic clients removing their own locks as part of a normal run. Separately, the server refuses to write a file that already exists, with or without the flag, so an existing pack file cannot be replaced.

For Borg, the documentation is careful to say that append-only "only affects the low level structure of the repository, and running borg delete or borg prune or reading from the repository will still be allowed." A prune by a compromised client is accepted, but because compaction is disabled the data stays in the segment files. Borg keeps a transaction log in append-only mode so a repository can be rolled back to an earlier state, provided borg compact has not been run since. Running borg init through a borg serve --append-only server does not make an append-only repository; borg init --append-only does.

How do you build it on S3?

S3 has no switch called append-only. You build it from what the credential is allowed to do:

  • Leave out both delete actions. The Service Authorization Reference lists s3:DeleteObject, described as removing "the null version of an object and insert[ing] a delete marker", and s3:DeleteObjectVersion, which removes "a specific version of an object". With versioning on, a plain delete only adds a delete marker, and a permanent delete needs the version ID and the second action. Without the first, the backup credential cannot even hide objects.
  • Say it twice. An explicit Deny overrides an Allow in identity and resource policies. A Deny on both delete actions in the bucket policy protects against a later, broader Allow added to the backup user.
  • Close the overwrite gap. Without versioning, uploading to an existing key overwrites it. With versioning on, Amazon states that an overwrite "results in a new object version", so the old one is kept. The other route is the If-None-Match header, which makes a write fail with 412 if the key exists. Amazon says bucket owners can enforce conditional writes by bucket policy.
  • Keep versioning for the recovery path. If an attacker overwrites keys with garbage, the previous versions are the way back.

Risk: an explicit Deny on both delete actions also blocks your own clean-up credential if the same policy covers it, and no Allow overrides it. Back out by removing the Deny statement (keep a copy first), then confirm clean-up works from the admin identity.

restic is a caution here. Its documentation names rest-server and rclone as the ways to serve an append-only repository and says "few other standard backends do". It documents no S3 policy for restic. Because rest-server's own code exempts lock files, a bucket policy that denies all deletion may stop restic removing its own lock files. restic's design reference says "at most one process can have an exclusive lock on the repository, and during that time there must not be any other locks", so leftover locks can make commands such as prune report the repository as already locked until the clean-up credential removes them. Test on a scratch repository first. rclone documents --append-only on rclone serve restic as "Disallow deletion of repository data", and its page does not discuss locks.

Where does clean-up run, and with which credential?

restic states the model plainly. Removing snapshots and recovering space with forget and prune needs full read, write and delete access, and "if an attacker has this, the protection offered by append-only mode is naturally void." Its recommended setup is "a separate and well-secured client whenever full access to the repository is needed." Borg's equivalent is two keys in authorized_keys: one with borg serve --append-only for the backup clients, and one without it for management. Borg's own example comment for the first is "key used for not-always-trustworthy backup clients".

There is a second trap, documented by restic. An attacker who can add snapshots can add garbage ones with timestamps just newer than yours, so that a policy such as "keep three weekly snapshots" removes the real ones. restic's advice is to use --keep-within in forget on append-only repositories. It adds a limit: if forget --keep-within 7d runs 8 days after the last good snapshot, the attacker can still remove all legitimate snapshots.

Example (illustrative): host fs01 sends 40 GB of new data a day to backup01, and nothing it stores is reused later. Retention is 30 days, so the steady store is 30 × 40 = 1,200 GB. Clean-up runs from a separate admin host every P days. Until then, expired data stays. The store peaks at about (30 + P) × 40 GB: monthly clean-up (P = 30) peaks at 2,400 GB; weekly (P = 7) at 1,480 GB; daily at 1,240 GB. A tool that never runs clean-up grows by 40 GB a day without limit. Each extra GB-month is billed at the provider's rate R. The calculation is rough: real stores share blocks and a day's data is rarely all new.

What does append-only not protect?

Threat Stopped by append-only? Why
Malware on the backed-up host deletes backups through the backup credential Yes The credential is refused
The same malware captures the repository password No restic's threat model says an attacker with append-only access could "capture the password and decrypt backups from the past and in the future"
New backups written after compromise No They are written by a host the attacker controls, so they are untrustworthy. restic says old snapshots made before the compromise remain restorable
Attacker steers a later forget into removing the real snapshots Only if you use --keep-within and run clean-up in time Garbage snapshots can crowd out good ones
Admin of the storage server or cloud account No They can delete the files directly, or change the policy
Clean-up credential stolen No It can delete everything
Storage failure No This is protection from deletion by credential, not from damage to the disks

If you need protection from the storage admin too, the answer is a lock the storage service enforces, such as Object Lock or an immutable attribute. That is a different control, with different costs.

How do I check this on my own system?

  1. rest-server. It prints the append-only status at start-up (changelog entry 295). Look in its log, or the service unit, for --append-only. Then check that the clean-up job does not use the same credential as the backup job.
  2. Borg. For each key in authorized_keys, check whether the command= entry includes borg serve --append-only, and whether the key used by the backup client is the one that does. For the repository setting, read it without changing anything with borg config /path/to/repo append_only. The setting is local-only: Borg says this command works only on local repositories.
  3. S3. Read the policies attached to the backup user, and the bucket policy, and look for s3:DeleteObject, s3:DeleteObjectVersion, wildcard actions such as s3:*, and whether versioning is on. A wildcard defeats the design.
  4. Ask your provider if the storage is run for you: Which credentials can delete? Who holds the clean-up credential and where is it kept? Can the same people who run backups also change the policy? How long does expired data stay before it is freed, and is that time in my bill?

Risk: to set Borg's append-only mode on a repository you run borg config /path/to/repo append_only 1. Borg says you can move between append-only and normal operation with the same command, so you can back out. Borg also warns that the repository config file is one of the few parts not versioned or backed up, so keep a copy first. From then on borg compact frees no space, so do clean-up through a separate, normal-mode key and plan the schedule.

Risks and limits

  • These commands have not been run or lab-tested. They are taken from vendor documentation and source code.
  • The rest-server behaviour is taken from the v0.14.0 source tag. The README on its main branch is ahead of that release, so check the version you run.
  • The Borg pages are the 1.4.5 documentation. A different major version may behave differently.
  • An S3 policy built from the action list above has not been tested against restic or any backup product. Locks and multipart uploads may need other permissions. Prove it on a scratch bucket first.
  • Append-only does not encrypt, verify or restore. A repository can be append-only and still unrestorable. Run restore tests.
  • If the machine being backed up is compromised, everything written after that moment is suspect, whether or not deletion was blocked.

Sources