Australian cloud for business software — built and run in Sydney since 2002

NetLeverage
NetLeverage

Compressed and encrypted files do not deduplicate: what that does to backup storage

Technical article
Backup & Recovery

About this article: This article is reviewed against the cited sources. This is source research, not lab validation. Check the stated versions and your own requirements before making changes.

A deduplicating backup keeps one copy of each repeated piece of data, so a file that changes a little costs a little. That only works when the backup tool can see the repeats. Compression and encryption applied before the backup tool reads the file hide them: one small edit to the original can change most of the bytes that come out, and the tool then stores most of the file again every time.

This is why "the backup tool deduplicates" and "my storage grows by nearly the full size of these files every night" can both be true. The fix is usually to move the compression or encryption to the backup tool, or to keep those files out of the main backup job.

Why do compression and encryption defeat deduplication?

Deduplication cuts data into chunks and stores each distinct chunk once. restic describes this: files are split "into variable length Blobs cut at offsets defined by a sliding window of 64 bytes", using Rabin fingerprints, with blobs of 512 KiB to 8 MiB and an average aim of 1 MiB. Its design reference adds that "for modified files, only modified Blobs have to be saved in a subsequent backup", and that this works even when bytes are inserted or removed in the middle of a file.

That description assumes unchanged parts of a file keep the same bytes. Compression and encryption break the assumption in different ways:

  • Compression rewrites the file as a stream of codes. How much of the output shifts after a small change depends on the format and the tool. This is an expectation, not a rule: a format that compresses each entry separately may keep untouched entries identical, while one continuous compressed stream may not.
  • Encryption is designed so the output shows no pattern from the input. SQL Server's own documentation says that "encrypted data compresses significantly less than equivalent unencrypted data", and a deduplication tool looking for repeated patterns has the same problem.

The practical test is simple: change a small part of the original, produce the compressed or encrypted version again, and compare how much new data your backup store takes in. Section "How do I check this on my own system?" shows how.

Where in the chain does the encryption happen?

The order matters more than whether encryption is used at all. Borg's internals page shows an encrypting backup tool that still deduplicates: it states that each chunk's key is id_hash(unencrypted_data) and that "compression is applied after deduplication". The identifier that decides whether two chunks are the same comes from the unencrypted data. restic likewise encrypts every blob in the repository ("AES-256 in counter mode") after cutting the file into blobs.

So encryption done by the backup tool, after it has cut and compared the data, does not by itself defeat deduplication. Compression and encryption done before the tool reads the file do.

Where it happens What the backup tool sees What to expect from deduplication
Backup tool encrypts and compresses its own store Original plain data Normal. Deduplication works on the plain chunks first (Borg documents this order).
Plain files, documents, uncompressed exports Original plain data Normal.
SQL Server BACKUP ... WITH COMPRESSION file A compressed stream Little to match between nights.
SQL Server backup with ENCRYPTION Ciphertext Little to match between nights.
TDE database, data files copied by a file-level or image-level backup Pages encrypted before they were written to disk Little to match for any page that changed.
Zip, archive and media files recreated nightly Compressed streams Depends on the format; assume little until you measure.
Whole volume encrypted below Windows (BitLocker) and read as raw blocks Ciphertext The first run after encryption is large. See "Turned on BitLocker and the next backup was a full copy: why, and how to plan the one-off growth".

TDE encrypts at the page level: Microsoft documents that "the pages in an encrypted database are encrypted before they're written to disk". A tool that copies the data files therefore reads encrypted pages. A tool that asks SQL Server for a native backup gets a backup file instead, which is encrypted with the same key hierarchy (Microsoft documents that backup files of TDE databases are also encrypted with the database encryption key).

What does SQL Server do about compression and encryption together?

SQL Server compresses backups on request (COMPRESSION) and can encrypt them (ENCRYPTION, with an algorithm and a certificate or key). The compression page lists what lowers the ratio: random data, one large row per page, encrypted data and an already compressed database.

For TDE databases there is a version detail. From SQL Server 2016, a MAXTRANSFERSIZE above 65536 enables an optimised path that "first decrypts a page, compresses it, and then encrypts it again". Without it, compression works directly on the encrypted pages "and might not yield good compression ratios". From SQL Server 2019 CU5, when a backup uses WITH COMPRESSION or the server default is on, MAXTRANSFERSIZE is raised to 128K automatically.

That improves the size of one backup file. It does not make tonight's compressed, encrypted file resemble last night's file. A smaller .bak can still cost more in a deduplicated store than a larger plain one, because the store cannot find the repeats in it.

What does it cost?

Example (illustrative): server fs01 holds 400 GB of ordinary files that change 1% a day (4 GB). It also holds a 300 GB folder of exports that is recreated every night with about 1% different content. The backup keeps 30 daily points and stores each chunk once. All figures are decimal GB, and the arithmetic ignores the compression the backup store itself applies.

Case A: the exports are compressed 3:1 before the backup runs, so the folder is 100 GB. Assume each night's compressed file shares almost no bytes with the night before.

  • Ordinary files: 400 GB for the first point, then 4 GB new on each of the next 29 points: 400 + (29 x 4) = 516 GB.
  • Exports: 100 GB new every night for 30 nights: 30 x 100 = 3,000 GB.
  • Total: 516 + 3,000 = 3,516 GB.

Case B: the exports are left uncompressed (300 GB) and the backup tool deduplicates them, with the same 1% change (3 GB a night).

  • Exports: 300 GB first, then 3 GB new on each of the next 29 points: 300 + (29 x 3) = 387 GB.
  • Total: 516 + 387 = 903 GB.

Compressing at the source made the stored total 3,516 / 903 = about 3.9 times larger, although every night's files are smaller on fs01 itself. The numbers depend on how much of each night's file really changes and on whether your tool matches chunks inside compressed data, so measure before you rely on them.

How do I check this on my own system?

Start by listing which file types hold the space. These commands are from Microsoft's PowerShell documentation; Get-ChildItem advises -LiteralPath with -Recurse rather than -Path:

Get-ChildItem -LiteralPath D:\ -Recurse -File | Group-Object -Property Extension -NoElement | Sort-Object -Property Count -Descending

That counts files by extension, the pattern shown in the Group-Object documentation. Look for .zip, .7z, .bak, .mp4, .jpg, .pst, database exports and any archive format your applications write.

For SQL Server, msdb.dbo.backupset records how each backup was written:

SELECT database_name, type, backup_start_date, backup_size, compressed_backup_size, key_algorithm
FROM msdb.dbo.backupset
ORDER BY backup_start_date DESC;

compressed_backup_size is the byte count of the backup stored on disk, key_algorithm shows the encryption algorithm (the documentation says NO_Encryption means the backup was not encrypted), and Microsoft gives backup_size / compressed_backup_size as the compression ratio. To see which databases use TDE, Microsoft's TDE page uses:

SELECT * FROM sys.dm_database_encryption_keys WHERE encryption_state = 3;

A value of 3 is described there as an encrypted state.

Then run a small controlled test with your own backup tool on a copy of one suspect file: back up, change a few lines in the source, regenerate the compressed or encrypted file, back up again, and compare how much the store grew. A grown store close to the full file size means the tool is not finding repeats.

Questions to put to a backup provider:

  • Does the tool compress and encrypt after it has cut and compared the data, or before?
  • How is "stored size" counted for a file that was already compressed or encrypted when the tool read it?
  • Can particular folders or file types be excluded from the main job, or given a separate retention?
  • Are encrypted databases protected by a native SQL backup, or by copying their data files?

What can I change?

Situation Option Cost or trade-off
Applications write compressed exports nightly Let the backup tool compress, and keep the export plain More space on the source disk; less in the backup store
SQL .bak files are compressed, then also copied by a server backup Keep only one of the two copies in the main job (see the SQL .bak article) Needs a plan for the other copy
Windows Data Deduplication is on for the volume Use its ExcludeFileType, NoCompress and NoCompressionFileType settings for archives and media Microsoft says "multimedia or files that are already compressed, do not benefit very much" from optimisation
Archives rarely change Accept the cost; each version is stored once Predictable, and it grows only when the file changes
Encrypted data must stay encrypted at rest Keep encryption, and budget for full-size storage of the changed files Security requirement, not a fault

Risk: Switching off encryption or compression on a source to improve deduplication changes how that data is protected on its own disk and may break a security or compliance requirement. Decide that with the data owner first. For SQL Server backups, an encrypted backup cannot be restored without its certificate or key, so changing the encryption setting also changes which certificate you must keep. There is no automatic way back from a backup written without encryption except taking a new encrypted backup.

Risks and limits

  • These commands have not been run or lab-tested. They come from the Microsoft PowerShell and SQL Server documentation and the restic and Borg design pages.
  • The statement that compressed or encrypted files give a deduplicating store little to match is an expectation from how chunk matching works. None of the cited sources measures it for a specific product, and some archive formats keep unchanged entries identical, so measure with your own tool.
  • restic's design reference does not state what is hashed to decide that two blobs are the same. Only Borg's page says the identifier comes from the unencrypted data.
  • The Windows Data Deduplication pages describe settings for already compressed files but do not mention encrypted files, so nothing here is claimed for them.
  • SQL Server behaviour is described for SQL Server 2017 to 2025 pages; SQL Server Express and Web editions do not support encryption during backup.
  • Figures in the example are made up to show the arithmetic.

Sources