Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Will keeping backups twice as long double the storage cost?

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Only if every restore point is stored as a full copy. If the backup stores each block once and later points add only what changed, doubling retention adds the new data from the extra period, not another copy of everything. In the worked example below, doubling retention from 30 to 60 days raises stored data by about 38%, not 100%. With weekly full copies the rise is about 82%. The answer depends on how the provider stores points, and that is a question you can put to them in plain terms.

Why does the answer depend on how points are stored?

A restore point is a view of the data at a moment. How much storage a view needs depends on the method.

  • A full copy at every point. Each point stores everything. AWS says that where a resource type does not support incremental backups, "each backup is a full copy, which can result in higher storage costs".
  • Full once, then only changes. AWS describes the incremental method: "The first backup of an AWS resource backs up a full copy of your data. For each successive incremental backup, only the changes to your AWS resources are backed up."
  • Only data new to the store. restic's design notes say that for a modified file "only modified Blobs have to be saved in a subsequent backup", and the same data from a second run is not stored again.
  • A mix. Azure's estimator describes SQL Server policies of "daily differentials with weekly/monthly/yearly full backups", and says the alternative of daily, weekly, monthly and yearly full backups "will consume slightly more storage".

Retention then decides how many of those units you carry. Each day you keep adds one more point's worth of new data, or one more full copy, or one more incremental, depending on the method.

What does doubling retention do to each method?

The table uses one server and the same two numbers for every row: 1,000 GB of protected data and 20 GB of new or changed data a day. It compares a 30-day window with a 60-day window, with one point a day.

Method Stored at 30 days Stored at 60 days Increase
Full copy at every point 1,000 x 30 = 30,000 GB 1,000 x 60 = 60,000 GB 100% (2.00 times)
Weekly full, daily changes 5 fulls + 25 daily = 5,000 + 500 = 5,500 GB 9 fulls + 51 daily = 9,000 + 1,020 = 10,020 GB 82% (1.82 times)
Only new data stored 1,000 + (20 x 29) = 1,580 GB 1,000 + (20 x 59) = 2,180 GB 38% (1.38 times)
Only new data, high change (100 GB a day) 1,000 + (100 x 29) = 3,900 GB 1,000 + (100 x 59) = 6,900 GB 77% (1.77 times)

Example (illustrative): a server fs01 holds 1,000 GB. The table's first three rows use 20 GB of new data a day. The last row shows what a busy database server might look like at 100 GB a day. Fulls fall on days 1, 8, 15, 22 and 29 in a 30-day window, which gives 5 fulls and 25 daily points. In a 60-day window they fall on days 1, 8 and so on to day 57, which gives 9 fulls and 51 daily points. In the only-new-data rows the first full copy is the oldest point, so 30 points hold 29 days of new data and 60 points hold 59. At a placeholder rate of R per GB-month, the only-new-data method costs 1,580 x R at 30 days and 2,180 x R at 60 days, which is 600 x R more every month. The full-copy method costs 30,000 x R more every month for the same extra 30 days. These are round numbers for arithmetic, not a measurement of any product.

Three things follow from the table.

  1. The method matters more than the retention period. The same policy change costs 38% or 100%.
  2. High daily change narrows the gap. At 10% a day (100 GB on 1,000 GB), the new-data method rises by 77%, nearer the full-copy figure than the 38% above. A store that keeps each block once saves less when most blocks are new each day.
  3. The weekly-full method is the easiest to under-estimate. Real products often keep an older full copy until every later point that depends on it has expired. The row above ignores that and so understates the stored size.

Does the bill double the day I change the setting?

No. Nothing ages out for the first 30 days after you extend retention, so stored data builds gradually. In the only-new-data row, it climbs from 1,580 GB to 2,180 GB over those 30 days, 20 GB a day.

How that shows up on an invoice depends on how the provider averages. AWS Backup says the storage billed in a month "is based on the average storage space used throughout the month (billed as GB-Month)". In its own example, 400 GB for 15 days and 800 GB for 15 days "= 18,000 GB-Days", which is 600 GB-Month. For the ramp above, the first month averages roughly 1,880 GB-months and each month after that 2,180 GB-months. Check whether the provider averages or uses a month-end figure, because the first invoice after a change looks different.

What else changes when retention gets longer?

Deleted data stays longer. Data removed from the server remains in older points until they expire. With 60 days of points, data deleted from the server today is still stored for another 60 days. If you delete large exports or old project folders to save space, longer retention delays the saving.

Clean-up runs against more points. restic's manual says forget removes points and prune removes the remaining data that was referenced only by the removed points. A longer window means each clean-up decides over more points, and the manual warns that pruning "can be a time-consuming process".

Minimum storage periods can add cost. Archive and cold classes charge a minimum period regardless of when you delete. AWS says cold storage backups "must be retained for a minimum of 90 days", Azure's Archive tier has "an Archive early deletion period of 180 days" and S3 Standard-IA charges for 30 days if an object is deleted sooner. Longer retention can move more data into these tiers, and the minimums then decide the bill. The archive-tier guide in the related list covers them.

Locks make a longer period one-way. In S3 Object Lock compliance mode, a locked version's "retention period can't be shortened". If you lengthen retention by applying a longer lock, going back later is not possible for the objects already locked.

How do I check this on my own system or with my provider?

Ask the provider, in these words:

  1. Is every restore point stored as a full copy, or is only new data stored after the first?
  2. If I double retention from 30 to 60 days, what is the expected extra stored size, in GB, for the servers I protect? Please show the sum.
  3. Is the bill based on a monthly average or on the end-of-month size?
  4. Are there minimum storage periods, and which tiers do they apply to?
  5. Are older full copies kept until later points that depend on them have expired?

On a system you run yourself with restic, the manual's stats command shows both sides: restic stats (the default restore-size mode) counts the size of the restored files, and restic stats --mode raw-data counts the stored blobs. The manual says comparing the two "can tell you how much deduplication is helping you". Before changing a policy, restic forget --dry-run with the new --keep-* options prints "what actions would be performed" without removing anything. Compare the stored figure after a month with the figure the formula predicted. If the measured rise is closer to the full-copy row than to the new-data row, the data is changing more than assumed, or each point is stored in full.

Risks and limits

  • Risk: shortening retention later deletes older restore points and cannot be undone once clean-up runs. Decide the retention against the recovery need, not only the invoice.
  • These commands have not been run or lab-tested. They are taken from restic's documentation.
  • The model assumes each day's new data is unique and stays until it ages out. Overlapping changes or rewrites of the same data make the real figure lower. Compression and metadata overhead are ignored.
  • Real products differ in how they keep older full copies, so ask rather than assume.
  • Prices and minimum periods change. Check the current terms of a quote before relying on any figure here.

NetLeverage includes a daily full disaster-recovery backup of each hosted server at no extra charge. Talk to an engineer.

Sources