About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.
Deduplication works by cutting data into pieces and storing each distinct piece once. The two common ways of cutting differ in what they do after a change. Fixed-size blocks cut at set offsets, so inserting a few bytes near the start of a file shifts every later block and makes them all look new. Variable-size blocks cut where the content says to, so only the pieces around the insert change. Which method a product uses changes how much it stores for some kinds of data, and the chunk size sets how large its index grows.
What is the difference between fixed and variable blocks?
Fixed blocks. The file is cut at even spacing. Borg's documentation describes its fixed chunker as one that "triggers (chunks) at even-spaced offsets, e.g. every 4MiB, producing chunks of same block size". It is simple and uses little processor time.
Variable blocks. The cut points depend on the data. Borg's buzhash chunker is "variable, content-defined blocksize" and uses a rolling hash. It cuts a chunk when "the last HASH_MASK_BITS bits of the hash are zero". The hash only decides where to cut. Borg's documentation states that deduplication relies on a cryptographic hash of the chunk contents.
restic does the same with another algorithm. Its design reference says "The implementation uses Rabin Fingerprints for implementing this Content Defined Chunking (CDC)." Microsoft's Data Deduplication on Windows Server also uses variable-size chunks: one of its listed steps is "Break files into variable-size chunks", where a chunk is "a section of a file that has been selected by the Data Deduplication chunking algorithm as likely to occur in other, similar files".
What happens to each after an insert?
Compare two edits to the same file.
| Edit to a 1 GiB file | Fixed 4 MiB blocks | Variable blocks (Borg defaults) |
|---|---|---|
| 100 bytes overwritten in place, in the middle | 1 block changes: 4 MiB stored again | 1 chunk changes: about 2 MiB on average |
| 100 bytes inserted at the start | All 256 blocks change: 1 GiB stored again | The chunk holding the insert changes: about 2 MiB on average |
| Data appended at the end | Only the final block and the new blocks | Only the final chunk and the new chunks |
The second row is the whole point. With fixed blocks, an insert moves every byte after it by the length of the insert, so every block that follows holds different bytes from last time. None of them match anything already stored. With content-defined cutting, the cut points follow the content. After the inserted bytes, the rolling hash soon sees the same content as before and cuts at the same places, so the later chunks are the same as last time.
No vendor page states this effect for fixed blocks in those words. It follows from how the documented methods work, and is the usual reason content-defined chunking exists. The size of the re-stored piece is a typical figure, not a guarantee: Borg's default chunks run from 512 KiB to 8 MiB, with an average near 2 MiB, and the hash looks at a window of 4,095 bytes.
Which edits actually occur depends on the data:
- Documents, source code, text, mail are often edited by inserting or deleting in the middle, which shifts what follows. Variable chunks suit them.
- Disk images and raw devices change mostly in place. A disk block is rewritten where it is, and nothing shifts. Fixed blocks lined up with the disk's own blocks can work well for that. Borg's own examples show
--chunker-params fixed,4194304for backing up a raw device and a sparse disk image. This part is reasoning from how disks work, not a Borg recommendation.
Is variable always better?
No. There are trade-offs on both sides.
- Variable chunking costs more processor time on the backup client, because it computes a rolling hash across the data.
- Its chunk sizes are not uniform, so the store handles chunks from 512 KiB to 8 MiB (Borg) or 512 KiB to 8 MiB with a 1 MiB average (restic).
- Fixed blocks have the lowest overhead: Borg describes its fixed chunker as having "low cpu overhead" and supporting sparse files, which its content-defined chunker does not.
Microsoft's own pages show the trade-off between two of its features. The page for ReFS deduplication in Azure Local says that feature "operates at the data block level and uses fixed block size depending on the system size". Its FAQ adds that "Windows Data Deduplication might provide better storage savings due to its use of variable block sizes", while ReFS deduplication is designed for active workloads and "focusing on minimizing performance impact after optimization". Both are deduplication. They are tuned for different jobs.
How does chunk size change the size of the index?
Every chunk needs an index entry, and the index is held in memory while the backup runs. A smaller average chunk finds more repeats and gives a larger index. A larger chunk gives a smaller index and stores more per change.
Borg documents the cost. Its memory estimate is chunk_count * 164 + total_file_count * 240 bytes, where chunk count is the total file size divided by the chunk size. At the default chunk size of 2 MiB (2^21 bytes), 1 TiB of data is 1 TiB ÷ 2 MiB = 524,288 chunks, which is 524,288 × 164 = 85,983,232 bytes, about 82 MiB. With fixed 4 MiB blocks it is 262,144 chunks, about 41 MiB. These figures leave out the file-count term.
Borg's own example shows the extreme: for 1 Mi files (2^20) with a total size of 1 TiB, its memory estimate is about 2.8 GiB with buzhash,10,23,16,4095 (an average near 64 KiB), against about 0.31 GiB with the defaults. A backup product that cuts very small chunks to chase a better ratio pays for it in memory.
restic's published sizes are "Blobs are of 512 KiB to 8 MiB in size", aiming "for 1 MiB Blob size on average", and "Files smaller than 512 KiB are not split". For restic, a small file is a single blob.
Does the chunk size or method affect the bill?
It can, in three ways, all through the stored GB the bill is based on:
- Space used after edits. The insert case above is the largest effect. A file server or a mail store with inserts and rewrites stores much less under variable chunks.
- Space used by the index. Smaller chunks mean a larger index. If the provider passes index and metadata space on to you, small chunks are not free.
- How much two backups share. restic's design reference says an irreducible polynomial "is selected at random and saved in the file config". So two restic repositories chunk the same file differently, and data in one cannot be matched against data in the other. That follows from the documented design. Check whether a provider's store is one shared store or many separate ones.
What does one inserted paragraph cost?
Example (illustrative): a file fs01\projects\plan.docx is 1 GiB (1,024 MiB). Two stores back up fs01 nightly. Store F cuts fixed 4 MiB blocks and Store V cuts content-defined chunks with a 2 MiB average. One night an editor adds a paragraph near the top of the file, which inserts about 100 bytes.
- Store F has 1,024 MiB ÷ 4 MiB = 256 blocks. Every block after the insert now starts 100 bytes later, so none match last night's. About 256 × 4 MiB = 1,024 MiB is stored again before compression.
- Store V has about 1,024 MiB ÷ 2 MiB = 512 chunks. One chunk, about 2 MiB, contains the insert. The rest match. About 2 MiB is stored.
The ratio between them is 1,024 ÷ 2 = 512 to 1. With a weekly edit of that kind and 52 weeks, Store F stores 52 × 1,024 MiB = 53,248 MiB, about 52 GiB, for one file's edits, against 52 × 2 = 104 MiB for Store V. The numbers show the shape of the effect, not what your data does. Real chunks vary in size, and a file that is edited in place would not show this gap.
How do I check this on my own system or with a provider?
Ask the provider:
- Is the block size fixed or variable? If fixed, what size?
- What is the average chunk size, and the minimum and maximum?
- Is deduplication done on the client before data is sent, or on the store after it arrives?
- Is the store shared across all your backups, or is each backup source separate?
On a Borg repository, you can see the real effect of an edit. After a backup, edit a copy of a test file and run another backup with statistics:
borg create --stats ::test-{now} /path/to/test-folder
Risk: this command writes a new archive into the repository, so the repository grows by what is new, and the archive stays in the restore-point list. To back out, delete that one archive by name with borg delete and run borg compact to free the space. Run it on a scratch repository if you do not want a test archive among real restore points.
The Borg documentation says the "This Archive" deduplicated size "is most interesting as that is how much your repository will grow". To see the chunker settings Borg uses by default, the create page lists --chunker-params with "default: buzhash,19,23,21,4095".
On restic, restic stats --mode raw-data shows blob data stored in the repository, so you can compare it before and after a test edit.
Risk: do not change chunker settings on a repository that holds live backups to experiment. Borg's documentation describes the parameters but does not say what happens to deduplication against data already stored. Test on a new scratch repository.
Read next
Related articles you may want next.
- Backup & RecoveryDeduplication ratio explained: what 10:1 on a backup report means for the billWhat a 10:1 deduplication ratio on a backup report means, why it climbs with retention, and how to turn it into GB stored per GB protected.Read article
- Backup & RecoveryBacking up and restoring Data Deduplication volumes: what is really on the diskPlan Windows Data Deduplication recovery around the chunk store, supported backup modes, compatible servers and realistic restore capacity.Read article
- Backup & RecoveryCompressed and encrypted files do not deduplicate: what that does to backup storageWhy compressed and encrypted files, SQL .bak files and zipped exports store at near full size in a deduplicated backup, and how to find them first.Read article
Risks and limits
- These commands have not been run or lab-tested. They are taken from Borg, restic and Microsoft documentation.
- The insert effect on fixed blocks is derived from the documented mechanism. None of the cited pages states it, and the numbers in the example are made-up round figures.
- Borg figures are from version 1.4.5 documentation. Borg 2.x and other restic versions may use other defaults.
- Microsoft's pages do not state the chunk sizes of Windows Data Deduplication, so no size is given here for it.
- A real backup product may combine methods, for example fixed blocks for disk images and content-defined cutting for files. Ask what is used for your data type.
- Compressed or encrypted files do not deduplicate well with either method. That is a separate cost question.
Sources
- Borg internals: data structures: Borg 1.4.5 documentation, no page date, accessed 17 November 2026
- borg create: Borg 1.4.5 documentation, no page date, accessed 17 November 2026
- borg delete: Borg 1.4.5 documentation, no page date, accessed 17 November 2026
- restic design reference: restic documentation 0.19.1, no page date, accessed 17 November 2026
- restic-stats(1), v0.19.1: restic project, accessed 17 November 2026
- Understanding Data Deduplication: Microsoft Learn, page date 2022-02-18, updated 2025-09-22, accessed 17 November 2026
- Optimize storage with ReFS deduplication in Azure Local: Microsoft Learn, page date 2026-05-14, accessed 17 November 2026
This article was originally published on .