Secrets services on the recovery critical path

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Do not start the systems that need secrets until the secrets service can actually answer, and do not store the keys that open it only inside that service. A sealed store can see its own disks and still be unable to decrypt them. A Vault Enterprise disaster-recovery secondary that has not been promoted will not serve ordinary clients. A snapshot restore is not finished when the command that accepted the file has exited.

The concrete behaviour below is HashiCorp Vault, from documentation independently reopened on 13 November 2026. The planning problem is wider than one product. NIST SP 800-34 Rev. 1 already says that if you encrypt backup media, the keying material has to be managed, ideally in one place, and stored separate from, but still reachable by, the encrypted data. Lose the keys and the backup is a sealed box. Vault is what that sentence looks like when the sealed box is also the thing your applications call for database passwords, tokens and encryption keys.

This article does not give commands to unseal, promote or restore. Those operations change who can read secrets, and they were not run here.

Why it matters

NIST defines a resource's recovery time objective as the time it can stay down before other resources and the business process are hurt. A secrets service has almost no job of its own. Its clock is the clock of every application, pipeline and administrator tool that cannot authenticate or decrypt without it. Restoring those applications first produces a queue of failures that look like bad backups.

There is a second, stricter failure. NIST's warning is about keys for the backup. Vault's warning is about keys for itself. If the mechanism that unseals Vault is gone, HashiCorp says the cluster cannot be recovered, even from backups. You can have a perfect snapshot and still have nothing a client can read.

How it works

A restart needs the configured unseal path

HashiCorp's seal document says a Vault server starts sealed. In that state it can access the physical storage and cannot decrypt the data on it. Before it is unsealed, the only operations are to unseal it and to check status. Applications that retry for a secret are not waiting on a slow disk. They are calling a process that is refusing to decrypt.

The layers are specific, and they matter when you decide what to put in an envelope.

• Vault encrypts data with an encryption key in a keyring, stored in the storage backend.
• That key is itself encrypted by a root key, stored with the data.
• The root key is encrypted by an unseal key.

Unsealing is how you obtain the root key so the rest can be decrypted. The default configuration does not hand one operator the whole unseal key. It splits the key with Shamir's secret sharing. Operators submit shares, in any order, until a configured threshold can reconstruct the key. With more than one node, each node must be unsealed with that threshold. Vault does not copy a partial unseal from one node to the others.

A node stays unsealed until somebody seals it, the server restarts, or the storage layer hits an unrecoverable error. A recovery that includes a reboot therefore includes another unseal. Sealing itself is a single call by one operator with root privileges, and HashiCorp describes it as a way to cut off access during an intrusion: without the shares, users cannot read the data. During an incident, the person who can seal and the people who hold the shares are different kinds of power. Write down both. A root operator can lock the estate out of its secrets faster than the threshold of share-holders can open it again.

Auto-unseal moves the dependency, it does not delete it

Shamir unseal is manual. HashiCorp says automated installers can start Vault and still leave the unseal as a human step, and that most users will prefer auto-unseal. Auto-unseal gives the unseal key to a trusted device or service, such as a cloud key service or a hardware security module. At startup Vault asks that service to decrypt the root key.

That is a lifecycle dependency, and the document uses the word strict. If the cloud key is unavailable, or if it is deleted before you have migrated the seal, you cannot recover the cluster until the mechanism is back. If the mechanism or its keys are permanently deleted, the cluster cannot be recovered, even from backups. A secondary cluster in Vault Enterprise can be given its own seal configuration, which reduces that risk, and HashiCorp still notes you can lose items that were not replicated, such as local mounts.

Auto-unseal also changes which keys you hold. Initialising with a hardware module or a cloud key service returns recovery keys, not unseal keys. Recovery keys authorise operations such as generating a root token. They cannot decrypt the root key, and HashiCorp says they are not enough to unseal Vault when the auto-unseal mechanism is not working. A runbook that says "we have the recovery keys, so we can always open the backup" is wrong for this configuration. The recovery keys are the quorum for administrative acts. The cloud key or the hardware module is what opens the data.

A disaster-recovery copy does not serve until it is promoted

Disaster-recovery replication is documented as a Vault Enterprise mechanism, not as behaviour of every Vault. In that mode, secondaries share the primary's configuration, policy, secrets such as key-value data and transit encryption keys, and the token and lease infrastructure. The point of sharing tokens and leases is so that applications can keep working against the old primary's identities when the secondary is promoted. Local secret engines, auth methods and audit devices are replicated to a disaster-recovery secondary. That is different from performance replication, where local mounts stay local.

Until the secondary is promoted and becomes the new primary, it does not forward service reads or writes. It is not a warm read-only API you can quietly point applications at "just to test". Pointing them there before promotion fails closed, which is safer than serving stale writes, and it will be reported as the secrets service being down.

Enabling replication rewrites the secondary's seal. HashiCorp's table is the planning constraint:

• If the secondary uses auto-unseal, its recovery configuration and recovery keys are replaced.
• If it uses Shamir, its seal configuration and unseal keys are replaced.
• In both cases the number of fragments and the threshold become the primary's.

The keys you wrote down when you first initialised the recovery-site Vault are not the keys that open it after it has been joined as a secondary. The shares that match the relationship you will actually fail over to have to be the ones in the envelope. When replication is enabled, performance standby nodes on the secondary seal themselves and have to be restarted to unseal. A promotion is not "the virtual machine is already running".

There is no requirement that both clusters use the same storage engine, and no requirement that they use the same seal type. Different seals are allowed, with a pointer to seal-wrapping details this article does not cover. Different seals do not mean the secondary keeps its old keys. The replacement rule above still applies.

Snapshots are a point in time, and the command returns early

For Vault's integrated storage (the Raft backend), vault operator raft snapshot save takes a snapshot you can restore to that point in time. HashiCorp is explicit that this does not apply to deployments that use Raft only for high availability in front of some other storage. A restore installs the snapshot and returns the cluster to the state defined in it. That is a rewind. Tokens, leases and secrets created after the snapshot are gone. Choose the snapshot with the same care as any other recovery point: what it contains, and what it will throw away.

The restore command's own warning should be in the sequence, not in a footnote. The command exits when Vault has validated and accepted the file, not when the restore has finished. The rest runs in the background: tear down subsystems, swap the database, start the subsystems again. HashiCorp says not to resume ordinary operations until the server logs show "post-unseal setup complete" and "vault is unsealed". A pipeline that treats a zero exit code as "secrets are back" will start applications against a Vault that is still swapping its database.

Recovery mode is not the disaster-recovery path

HashiCorp documents a separate recovery mode, started with a flag, for a Vault that will not start because of a bug. The purpose is direct access to storage. The same page says this probably will not help unless a Vault expert is advising.

In that mode the usual subsystems do not run, including expiration, clustering and calls from other nodes. You do not unseal in the normal way. You generate a recovery token, and requests go to a raw storage path authenticated with that token. The token is not stored. If you restart, you generate another, and only one such token exists at a time. With integrated storage, recovery mode resizes the Raft cluster to one node, because Raft will not accept changes without a quorum and this mode intends to change data from a single node. Returning to service requires the vendor’s controlled cluster-reformation procedure, with the surviving data protected and membership reconciled. Destructive node-cleanup and peer-list instructions are deliberately outside this planning article. If integrated storage is only the high-availability layer, recovery mode cannot change the Raft data. It can change the physical data of the real storage backend, and the Raft notes do not apply.

Our recommendation: do not put recovery mode in the ordinary estate sequence. Catastrophic loss of a cluster is what disaster-recovery replication is for, and a snapshot restore is the integrated-storage rewind. Recovery mode is surgery on a cluster that will not boot, it discards the cluster quorum by design, and the vendor tells you to have an expert in the room. A runbook step that says "if Vault looks down, start recovery mode" will turn a seal problem into a split cluster.

Where this sits in the estate order

NIST's sequence rule is that components come back in an order that protects the downtime limit of the processes that depend on them. For secrets, our recommendation is:

1. Confirm the unseal path before you need it. Either enough Shamir shares are in the hands of people who can reach the recovery site without logging into Vault, or the auto-unseal service is reachable from that site and its key still exists. Shares stored as a secret inside Vault are not a plan. The server cannot decrypt them while it is sealed, which is the moment you need them.
2. Decide which cluster is allowed to become primary if you have Enterprise disaster-recovery replication. Do not send clients to the secondary before it is promoted. Verify node state and readiness after promotion; the documented automatic sealing of performance standbys occurs when enabling replication, not as a universal promotion step.
3. If you must rewind, restore a snapshot only on integrated storage, only to a chosen point, and wait for the unsealed log lines. Do not start consumers on the command's exit code.
4. Then start the consumers, in the business order from the impact analysis. An application whose first act is to fetch a database password will not get further than this gate.
5. Keep recovery mode off the default path. Use it when Vault will not start and someone who knows this product is directing the raw storage edits, then rejoin the other nodes deliberately.

A circular case belongs in the plan. The auto-unseal service might itself need a network, a name and an identity that you are still recovering. If the only key that opens Vault is temporarily unavailable in a failed region, the backup cannot be opened until that mechanism returns or an independently sealed recovery cluster is available. Permanent deletion is different: HashiCorp says the original cluster cannot then be recovered even from backups. The mitigation they name is control over that key, and, for Enterprise, a secondary with its own seal. Both have to be built before the disaster. They cannot be improvised after the key is deleted.

What to choose and why

These are our recommendations.

Risks and limits

• The product detail is Vault, current documentation, v2.x selector, independently reopened 13 November 2026. Disaster-recovery replication is an Enterprise document. Shamir seal, auto-unseal warnings and recovery mode are in the general concepts. Another secrets product will not share these state machines. The NIST rule about keys travelling separately from ciphertext still applies.
• Recovery keys and unseal keys are different objects once auto-unseal is on. Mixing the names up is how a carefully stored envelope fails at the moment of use.
• Enabling replication replaces the secondary's keys. A test that joined the secondary last year, and a runbook that still quotes the original initialisation ceremony, are describing a seal that no longer exists.
• A snapshot restore rewinds the cluster and the client command returns before that rewind finishes. Resuming early is an application incident on top of a storage operation.
• Recovery mode resizes integrated storage to one node and turns off clustering. It is not a standby you can enter casually.
• No command was run. This article withholds the command lines on purpose. HashiCorp's own recovery-mode page says the technique probably needs a Vault expert. A copied procedure without that person is how a quorum gets destroyed.
• Not covered: seal wrapping, performance-replication mount filters, and any cloud provider's key-deletion grace period. Those change the outcome and need their own current page.

Research and verification

Drafted with AI assistance on 11 November 2026 and independently source-reviewed on 13 November 2026. No article commands or recovery operations were executed. This is educational source research, not a lab-validated production procedure.

Sources

Independently reopened 13 November 2026.

• NIST Special Publication 800-34 Revision 1, May 2010: PDF. Keying material stored separate from encrypted backup data; RTO as impact on other resources.
• HashiCorp, Vault Seal/Unseal, v2.x documentation as served that day. Sealed state, Shamir shares, auto-unseal dependency, recovery keys.
• HashiCorp, Vault Recovery mode. Schema dateModified 15 April 2026. Direct storage access, recovery tokens, Raft resized to one node.
• HashiCorp, Vault Enterprise replication. Disaster-recovery secondaries, promotion, seal replacement on enable.
• HashiCorp, vault operator raft. Snapshot save and restore, including the early return of the restore command.

This article was originally published on 13 November 2026.