Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

LUKS on physical Linux servers: header backups, key slots and automatic unlock after a restore

Technical article
Backup & Recovery

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Normal LUKS recovery needs both usable header/keyslot information and a working unlock method on the recovery hardware. A separately preserved raw volume key is an exceptional recovery path, with severe security implications, rather than a substitute for planned key management. The cryptsetup FAQ puts the first risk bluntly: "if anything damages the LUKS header or the key-stripe area then decrypting the LUKS device can become impossible. This is a frequent occurrence." The second risk arrives with automatic unlock. A volume that opens itself at boot through a TPM or a network server can stop opening on replacement hardware, and a team that has not typed a passphrase in two years may find nobody knows it.

Plan for both: keep a protected header backup for each volume, keep a human-usable recovery key in escrow, and decide in advance which automatic unlock methods a restored system should and should not inherit.

Why it matters

Encryption at rest changes what "the disk survived" means. With an unencrypted server, an intact disk is readable by any recovery system. With LUKS, an intact disk is readable only by someone who can satisfy one of its key slots, and only while the header that describes those slots is intact.

The FAQ explains why the header is so fragile: the critical components for decryption are the salt values in the key-slot descriptors and the key slots themselves, and "for LUKS1, these are right in the first sector". Software that writes a partition boot sector, an accidental format, or a disk-imaging mistake that shifts a partition can destroy them. The FAQ adds that "in order to make a key-slot completely unrecoverable, changing about 4-6 bits in random locations of its 128kiB size is quite enough."

Automatic unlock adds another recovery dependency. Unattended boot is a real operational need, so many physical servers bind a key slot to the machine's TPM or to a Tang server on the local network. Those bindings are deliberately tied to a specific machine or network. That is the point of them, and it is also why they can fail after a hardware replacement.

How it works

The volume key and the key slots

LUKS encrypts data with a single volume key. Passphrases and other unlock methods do not encrypt the data directly. Each one opens a key slot, which holds a copy of the volume key protected by that method. The luksDump page warns that "the volume key cannot be changed without reencryption and can be used to decrypt the data stored in the LUKS container without a passphrase and even without the LUKS header."

Three consequences follow for recovery planning.

  • Changing or removing a passphrase changes a key slot, not the volume key. Old copies of the header still contain the old slots.
  • Losing every working key slot can make intact data inaccessible unless another valid header copy or the original volume key survives.
  • An image-level copy of the encrypted device carries the same header and volume key. A file-level restore into a newly created LUKS volume has a new volume key, so header backups of the old volume are of no use for the new one.

LUKS1 and LUKS2 headers

The FAQ notes that LUKS2 "is less vulnerable to corruption and has a 2nd copy" of its header, stores much of its metadata as JSON, and supports up to 32 key slots. It also says that after a quick format, LUKS2 "may still be recoverable without that header backup, for LUKS1 it is not." The second copy improves the odds against small accidents; it does not protect against the whole start of the device being overwritten, and it is no substitute for a backup kept elsewhere.

LUKS2 also holds tokens: metadata that tells unlock tools which method a key slot belongs to. TPM2, FIDO2 and PKCS#11 enrolments made with systemd-cryptenroll, and Clevis bindings, store their details in the header. Restoring a header therefore restores those bindings too.

What a header backup is, and what it gives away

The luksHeaderBackup page describes the command as storing "a binary backup of the LUKS header and keyslot area". The FAQ recommends it over copying bytes by hand because it "protects also against errors when non-standard parameters have been used in LUKS partition creation", and says a header backup "only needs an update if you change passphrases".

The same page gives the warning that matters for security: "The backup file and a passphrase valid at the time of backup allow decryption of the LUKS data area, even if the passphrase was later changed or removed from the LUKS device." It continues: "with a header backup, you lose the ability to wipe the LUKS device securely by just overwriting the header and keyslots. You must either securely erase all header backups or overwrite the encrypted data area." The FAQ repeats the point under secure erasure: "A LUKS header backup or full backup will still grant access to most or all data, so make sure that an attacker does not have access to backups or destroy them as well."

In practice, a header backup should be treated as key material, not as an ordinary configuration file. Store it encrypted, outside the server, with the same access controls as your recovery keys. Record which passphrases and tokens were valid when it was taken. If you revoke a passphrase because someone left the organisation, the revoked passphrase still opens every older header backup, so replacing a managed backup does not revoke an old copy already retained elsewhere. Account for image backups, retention and any escaped copies; a compromised volume key can require data re-encryption.

Restoring a header

The luksHeaderRestore page is explicit about scope: "All keyslots will be replaced; only the passphrases from the backup will work afterward. This command requires that the volume key size and data offset of the LUKS header and backup match." Restoring is therefore a deliberate rollback of every key slot and token, not a repair of one slot.

The FAQ gives a safer first step. A header file can be tested without writing anything to the device, by using the backup as a detached header in a controlled recovery environment. "If that unlocks your key-slot, you are good." It also advises: "If you are unsure about a header to be restored, make a backup of the current one first!" If the damaged device holds data you cannot afford to lose, the more cautious route is to image the device and test the header against the image.

TPM-bound unlock

The systemd-cryptenroll manual enrols a TPM2 chip into a LUKS2 key slot. A matching /etc/crypttab line then unlocks the volume at boot with tpm2-device=auto, as described in crypttab(5). The enrolment can be bound to Platform Configuration Registers (PCRs), which record measurements of firmware, boot loader and configuration. According to the manual, if --tpm2-pcrs= is not given, the enrolment binds to no PCRs at all; administrators who want measured-boot protection choose which ones to bind or use a supported signed-PCR policy. A direct enrolment with no PCR binding does not itself restrict which measured boot state can request unlock; other configured controls such as a PIN must be assessed separately.

The manual's PCR table shows why those choices matter in recovery:

  • PCR 0 holds core firmware code and "changes on firmware updates".
  • PCR 1 holds platform configuration, "typically contains serial and model numbers", and "changes on basic hardware/CPU/RAM replacements".
  • PCR 7 records Secure Boot state, changing "when UEFI SecureBoot mode is enabled/disabled, or firmware certificates (PK, KEK, db, dbx,...) changes".

The key in a TPM-bound slot is sealed by that physical TPM. On a replacement server, the new TPM cannot unseal a key sealed by the old one, so a TPM-bound slot does not travel with the disks. Even on the same server, a motherboard replacement, firmware update or Secure Boot change can alter bound PCRs and stop unattended unlock. Whether boot offers another unlock prompt depends on the initramfs and configuration. The recovery plan must provide a usable passphrase or recovery-key path even when unattended boot stops without a prompt.

systemd-cryptenroll --recovery-key creates a computer-generated recovery key. The manual describes it as "mostly identical to passphrases", but with "a guaranteed high entropy" and a character set "easy to type in". Enrolling one alongside a TPM slot, and escrowing it, is a direct answer to the replacement-hardware problem.

Network-bound unlock with Clevis and Tang

Tang is "a server for binding data to network presence": data is available "only when the system containing the data is on a certain, usually secure, network". The client stores some small metadata at provisioning time, then performs an HTTP POST to the Tang server to recover the key. Tang keeps no per-client secret; the README contrasts it with key escrow, which is stateful and "must be surrounded by a sophisticated backup policy". Clevis is the client side; clevis luks bind binds a LUKS device to a Tang server, a TPM or another pin.

Network binding has the opposite recovery profile to TPM binding. It does not care which hardware the disk sits in, so a moved disk can unlock on compatible replacement hardware when the required Tang exchange keys, client binding, boot integration and complete policy remain usable. Network reachability alone is insufficient. That is convenient for recovery and is also a risk: a restored encrypted copy satisfying the same Tang policy may open without a person. Decide whether restored copies in a test environment, a second site or a forensic lab should be able to reach production Tang servers.

Tang servers have their own recovery dependency. The README tells operators to rotate keys periodically, disabling advertisement of the old key first and deleting it only "after some reasonable period of time". If required Tang keys are lost, bindings that depend on them cannot use that Tang path; other LUKS slots or a threshold policy may still permit recovery. Back up the Tang key directory, protect it as key material, and do not delete rotated keys until every client has been rebound.

Combining methods

Clevis supports Shamir Secret Sharing, which splits a key into pieces encrypted by different pins and recovers it when at least a threshold t of them can be decrypted. The Clevis README shows a policy that uses both TPM2 and Tang. A threshold policy can require both the original machine and the trusted network, or accept either one. That choice is a security decision, but it also fixes in advance what will happen after a restore.

What to choose and why

Inventory each encrypted volume

For each volume, record its LUKS version, which key slots are in use and what kind they are (passphrase, recovery key, TPM2, Tang, FIDO2), and which PCRs any TPM slot is bound to. Store this outside the server. cryptsetup luksDump shows the header information without the volume key unless you explicitly ask for it, and the manual warns against --dump-volume-key because it exposes the key itself.

Keep a human path that does not depend on the old machine

At least one slot on every volume should be usable by a person on any hardware: a strong passphrase or a recovery key, escrowed with access controls and a named custodian. Test it on a schedule, because the people who set up automatic unlock are often not the people who will be asked for the passphrase years later.

Keep header backups, and treat them like keys

Take a header backup when a volume is created and again whenever slots change. Store it encrypted and separately from the server, record the date and the slots it contains, and destroy superseded copies when a revoked passphrase must stop working. Remember that header backups and full image backups of encrypted devices defeat a decommissioning approach that relies on wiping the header.

Decide how restored systems should unlock

Restore situation TPM-bound slot Tang-bound slot Passphrase or recovery key
Same server, disks reinserted Works unless bound PCRs changed May work if matching keys, binding and boot integration remain available Works if the matching slot, secret and header survive
Same server after firmware or Secure Boot change May fail, depending on bound PCRs May work if matching keys, binding and boot integration remain available Works if the matching slot, secret and header survive
Replacement server, same disks or image Fails: new TPM May work if matching keys, binding and boot integration remain available Works if the matching slot, secret and header survive
File-level restore into a new LUKS volume Must be re-enrolled Must be re-bound New slots needed
Image restored in an isolated lab Fails Fails unless lab reaches Tang Works if the matching slot, secret and header survive

(our reading, from the sources above)

After a recovery to new hardware, enrol the new TPM or re-bind to Tang, confirm unattended boot works, then retake the header backup, because the slots have changed. Remove slots tied to hardware you no longer own, using the slot-wiping options in the systemd-cryptenroll or cryptsetup documentation once you have confirmed a working replacement slot.

Recovering from a damaged header

If a volume reports that it is not a LUKS device, or every passphrase fails, stop writing to it. Make an image of the device first, then test your header backup as a detached header against the image, as the FAQ describes, before you restore anything to the original device. The FAQ's manual byte-copy method for LUKS1 headers exists, but it relies on knowing the header size, and it warns: "Do not change the filesystem if you were unable to determine the header size!" Without a header backup and with damaged salts, the FAQ's position is that "nothing (in the cryptographically strong sense) can be done". At that point the data backup is your recovery.

Risks and limits

  • LUKS1 header loss can be fatal. Overwritten LUKS1 salts can prevent recovery without another usable header, keyslot or separately preserved volume key.
  • Header backups weaken revocation. An old header plus an old passphrase decrypts the volume. Inventory and destroy superseded copies when revocation matters.
  • Automatic unlock is an access path. Tang-bound copies may open wherever the full binding policy can be satisfied; TPM-bound volumes rely on boot measurements you chose. Neither replaces a reviewed key-management policy.
  • Hardware replacement breaks TPM slots by design. Plan the human recovery path before the hardware fails, not after.
  • Distribution differences. Initramfs tooling, Clevis packaging and default crypttab options differ between distributions, and the PCR defaults described here are those of systemd-cryptenroll, not of every tool.
  • This is not a tested procedure. The commands named come from the documentation; none were run for this article. Rehearse header testing and unlock on non-production volumes or images before you depend on them.

Sources