Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Object storage bill for backups: request charges and minimum object sizes

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

Backups stored as objects in S3, Azure Blob Storage or Google Cloud Storage are billed for more than the gigabytes they occupy. Every upload, listing and read is a request with its own charge, and some storage classes bill each object as if it were a minimum size, however small. A backup tool that writes millions of small objects pays more on both counts than one that writes fewer, larger ones.

The size of the objects your backup software chooses is therefore a cost setting, not a technical detail. You can measure it before committing, and you can ask the provider and the software vendor three questions that put quotes on the same footing.

What is charged besides gigabytes?

Each provider lists several meters. The table below uses the structure of the providers' own pages. It quotes no prices.

Meter What triggers it in a backup Who controls it
Storage per GB-month Data held, including old versions Retention, versioning, the tool
Requests (writes, lists, reads) Every object uploaded, every listing, every read during a check or restore Object size and tool behaviour
Retrieval per GB Reading from infrequent-access or archive classes Restores and verification
Transition requests Lifecycle moving objects into another class Lifecycle rules, object count
Minimum storage duration Deleting or overwriting too soon Class choice, retention
Minimum billable object size Small objects in some classes Class choice, object size
Data transfer out Restores across regions or to the internet Measuring cloud egress before you commit is a separate topic

The S3 pricing page says: "You pay for requests made against your S3 buckets and objects. S3 request costs are based on the request type, and are charged on the quantity of requests as listed in the table below." It adds that browsing in the console also bills: "When you use the Amazon S3 console to browse your storage, you incur charges for GET, LIST, and other requests that are made to facilitate browsing." Azure's cost page lists Operations as a meter priced "per transaction", and Google's pricing page splits operations into "Class A, Class B, and free".

Minimum storage periods (30 days for infrequent-access classes, longer for archive classes) are charged when you delete early. Archive storage tiers have their own topic, which covers those periods and retrieval delay, so they are not repeated here.

Why does object size change the number of requests?

A backup of a given size is split into objects. The number of write requests is the data size divided by the object size, plus overhead. Halve the object size and the writes double.

Backup tools differ. Some write one object per file, some pack many small pieces into larger "pack" files, and some let you set a target size. restic, for example, documents a global --pack-size option: "set target pack size in MiB, created pack files may be larger". Ask your tool vendor what size its objects are and whether you can change it.

Listing matters too. S3's ListObjectsV2 returns up to 1,000 keys per request ("By default, the action returns up to 1,000 key names"), so a tool that lists the whole store before each run needs one request for every 1,000 objects, every time. S3 charges listings at the write rate: "LIST requests for any storage class are charged at the same rate as S3 Standard PUT, COPY, and POST requests." The same page says "DELETE and CANCEL requests are free."

Example (illustrative): backup01 writes 1 TiB of new backup data in a month into a bucket that holds 12 TiB in total. The tool lists the whole bucket once a night, 30 nights a month. 1 TiB is 1,048,576 MiB.

4 MiB objects 64 MiB objects
Write requests (1,048,576 / object size) 262,144 16,384
Objects in the 12 TiB bucket 12 x 262,144 = 3,145,728 12 x 16,384 = 196,608
List requests per night (objects / 1,000, rounded up) 3,146 197
List requests per month (x 30) 94,380 5,910
Total requests per month 356,524 22,294

The 4 MiB layout makes about 16 times as many requests (356,524 / 22,294 = 15.99). If R is the charge per 1,000 requests and lists and writes are priced alike, as on S3, the month costs 356.524 x R against 22.294 x R. The stored gigabytes are identical. Real tools add reads, checks and retries, so treat this as a floor.

What does a minimum billable object size do?

Some classes bill a floor per object. For S3, the pricing page says: "S3 Standard-IA and S3 One Zone-IA storage have a minimum billable object size of 128 KB. Smaller objects may be stored but will be charged for 128 KB of storage at the appropriate storage class rate." The S3 classes table also lists 128 KB for S3 Glacier Instant Retrieval. A million 16 KB objects in such a class are billed as a million 128 KB objects: eight times the space (128 / 16 = 8), which gigabyte totals in the provider's console will not reveal.

The archive classes add overhead instead. AWS states that S3 Glacier Flexible Retrieval and Deep Archive each need 40 KB of extra metadata per archived object: "32 KB of metadata charged at the S3 Glacier Flexible Retrieval rate" plus "an additional 8 KB data charged at the S3 Standard rate" (the Deep Archive note is identical with its own rate). A 10 KB object therefore carries four times its own size in overhead.

S3 Intelligent-Tiering has a different rule: "Monitoring and automation fees per object apply", and objects under 128 KB "are not monitored and always stored in the Frequent Access tier".

Lifecycle rules interact with this. AWS states that "the default behavior prevents objects smaller than 128 KB from being transitioned to any storage class", and the pricing page warns: "There are per-request ingest charges when using PUT, COPY, or lifecycle rules to move data into any S3 storage class. Consider the ingest or transition cost before moving objects into any storage class." Moving millions of small objects can cost more in transition requests than the lower storage rate saves.

Risk: moving or deleting objects to cut cost can add charges instead. A transition request is billed per object, and objects deleted, overwritten or moved before a class's minimum duration still incur the pro-rated charge. Test a lifecycle rule on one prefix first, and note that an expiry cannot be undone once an object is gone.

Azure and Google differ on size. Google's classes page states "No minimum object size." The two Azure pages cited here list minimum days (cool 30, cold 90, archive 180) but no minimum billable object size.

Do the units differ between providers?

Yes, and this is where quotes mislead. A price "per 10,000" is not comparable with a price "per 1,000".

Provider Request unit in its pages Classes Minimum billable object size
Amazon S3 per 1,000 requests (PUT, COPY, POST, LIST; GET, SELECT and all other; lifecycle transitions; data retrieval) By request type and storage class 128 KB in Standard-IA, One Zone-IA and Glacier Instant Retrieval; 40 KB overhead in Flexible Retrieval and Deep Archive
Azure Blob Storage "In most cases, it's the price of 10,000 operations" Operation types, by tier None listed on the two Azure pages cited
Google Cloud Storage per 1,000 operations Class A, Class B, free "No minimum object size"

Azure adds a warning that matters for backups: "The pricing pages don't list a price for each individual operation but instead lists the price of an operation type." You must trace each logged operation to its type before you can price it. Azure also notes that some operations, deletes and metadata calls among them, "may be charged on hot tier only, even if the operation is executed against objects on other tiers".

Google counts uploaded parts of an XML API multipart upload as stored data "until the multipart upload is either completed or aborted", and S3's CloudWatch NumberOfObjects includes parts of incomplete uploads. Clean-up of incomplete multipart parts is covered under large uploads that resume.

How do I check this on my own system or with a provider?

  1. Work out your average object size. In S3, CloudWatch publishes BucketSizeBytes once a day and NumberOfObjects. Divide bytes by objects. S3 Storage Lens has a free "Average object size" metric. Its object-count bands, such as objects from 0 KB to 128 KB, are in the Advanced tier, so they may cost extra.
  2. Count requests. In S3, CloudWatch has request metrics; on Azure, resource logs record each operation. For any provider, ask which report breaks last month's requests out by type.
  3. Ask the backup vendor: what is the object size, can it be set, does the tool list the whole store each run, and how often does it read objects back for checks?
  4. Ask the provider: what is the unit of each request price, which requests are free, is there a minimum billable size in the class I will use, and what overhead applies per object?
  5. Ask for restore costs: retrieval per GB, and the data-transfer charge for reading data out.

Risks and limits

  • The steps above give metric names and questions, not commands. Nothing here has been run or lab-tested, and metric names can change.
  • Prices are not quoted because they change and differ by region and class. Calculations use R as a placeholder.
  • The example counts only writes and one listing a night. Retries, reads for checks, restores and lifecycle actions add more.
  • The Azure and Google pages cited do not say which operations a given backup tool performs. Check with the tool.
  • Pricing pages, class lists and minimums change. Read the current page for your region and class before you decide.
  • Other S3-compatible providers may define request and minimum-size rules differently.

Sources