Kubernetes etcd Backup and Restore: What It Holds and How to Restore a Control Plane

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

An etcd snapshot restores the Kubernetes API objects and nothing else. To recover a control plane you also need the certificates and any encryption configuration. Permanent quorum loss is a principal disaster-recovery case. An intentional state rollback is a separate decision because it discards newer changes; every API server must be stopped before restoring etcd. Restore every member from the same snapshot, and use a revision bump so that controllers do not act on stale caches.

The commands below were not run in a lab, and are reference material, not a ready-to-run procedure, until a qualified engineer has checked them on a copy of the cluster. Hosts, paths and names are examples. A restore replaces cluster state, so rehearse on a cluster that does not matter.

Recommendations and interpretations are editorial unless attributed; source research is not lab validation.

Why it matters

Kubernetes uses etcd as its backing store for all cluster data. If most members are permanently lost, the cluster cannot change its state. Scheduled pods may keep running, but no new ones can be scheduled.

This article assumes you operate etcd yourself, for example on a kubeadm cluster. On a managed service the provider runs etcd, so this procedure does not apply. Check that provider's documentation.

How it works

What a snapshot contains

etcd is a multi-version key-value store. A transaction that modifies the keyspace advances its revision; one transaction can change multiple keys. The API server stores its objects under a key prefix that defaults to /registry.

A snapshot holds the keyspace at one moment, and it stops at the revision it was taken at. That split matters more than anything else in this article:

So back up the CA files and sa.key and sa.pub alongside the snapshot, and protect them at least as carefully.

A snapshot is sensitive

Kubernetes says Secrets are stored unencrypted in etcd by default. A snapshot file from such a cluster therefore contains them in readable form. Encrypt it, restrict who can read it and keep it away from the cluster it protects.

If you did enable encryption at rest, the key becomes part of the backup. Kubernetes warns that if you lose all copies of the key, you would need to delete the resources encrypted under it. A snapshot without its key is of little use for those resources.

Quorum

etcd tolerates up to (N−1)/2 permanent failures in a cluster of N members. Three members survive one loss and five survive two. Lose more than that and the cluster cannot accept writes until a majority returns, or until an operator starts disaster recovery.

What to choose and why

First decide whether a restore is the right fix

A restore rolls the whole cluster back, so it is often not the answer.

Taking the backup

Kubernetes documents two methods: the built-in snapshot and a volume snapshot. Use the built-in snapshot unless you have a specific reason not to.

sudo etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=<cert-file> --key=<key-file> \
  snapshot save /var/backups/etcd/snapshot.db

The certificate and key paths come from the description of the etcd pod. Kubernetes says creating the snapshot will not affect the member's performance.

Avoid copying member/snap/db as a routine backup:

• Copying that file may lose data that is still only in the write-ahead log.
• A copied file has no integrity hash, so restoring it needs --skip-hash-check.

Then record what you have:

etcdutl --write-out=table snapshot status /var/backups/etcd/snapshot.db

The output gives the hash, revision, total keys and size. A file that reports these has been read, but it has not been restored. Only a restore proves that.

Use etcdutl for status and restore. The etcd changelog records that etcdctl snapshot status and snapshot restore were removed in v3.6. The Kubernetes page still shows the older form as deprecated, so older runbooks may carry it.

Prepare the tools before any incident. A kubeadm node runs etcd as a container, and while etcd is stopped for recovery that container is unavailable, so there is nothing to run them inside. Install matching etcdctl and etcdutl on each control plane node, or keep them with the backups. Both are in the etcd release archive, and the install page shows them in the etcd container image.

Snapshot often enough that the data you could lose is acceptable, and take one before every upgrade. The schedule must support your recovery point objective, allowing for failed runs and copy completion; snapshot frequency alone is not a proven recovery point.

Version matching

Kubernetes says etcd restores snapshots from the same major.minor version, and that a different patch version is supported. Record the etcd version with each backup, and use a restore tool of that version. Kubeadm 1.37 defaults to etcd 3.7.0-0, while the etcd links here are to v3.6 pages. The v3.7 disaster recovery page matched the v3.6 text when we compared them. Check which etcd you are running.

Revision bump: why a plain restore is not enough

After a restore, clients may see the revision go backwards by hundreds or thousands. The etcd project calls this particularly a problem for Kubernetes, because controllers use watch-driven caches that may not refresh correctly. It highly recommends a revision bump for Kubernetes.

Two options work together:

• --bump-revision adds a number to the snapshot's revision, so revisions never decrease.
• --mark-compacted marks those revisions as compacted, which terminates watches and invalidates informer caches. It is required when the bump is above zero.

Size the bump to exceed the writes since the snapshot. etcd's example is 1,000,000,000, which it says covers a week-old snapshot below 1,500 writes per second. The arithmetic is 7 days × 86,400 seconds × 1,500, or about 907 million.

The revision rises with each change, so two readings from etcdctl endpoint status taken some time apart give you your real write rate. These options arrived in etcd 3.5.10.

Plan a three-node kubeadm recovery

Kubernetes requires all API servers to stop before every etcd instance is restored, followed by API-server restart. It also recommends restarting the scheduler, controller manager and kubelet to avoid stale state. This article does not provide manifest moves or data-directory replacement commands: those operations require a validated procedure for the installed topology.

The engineer's recovery plan should cover six gates.

Establish isolation and authority. Identify every old member, API server and management automation that could restart them. Keep the original cluster separated from the replacement. A new cluster identifier is not a substitute for network and process isolation. Record which snapshot was selected, its age and the changes that will be discarded.

Preserve the old evidence. Retain the original data directories, manifests, certificates, encryption material and logs before approving replacement. The storage location must have enough capacity for both old and new data, and the rollback plan must say which copy is authoritative. Never reuse a directory merely because its name resembles the expected default.

Specify the new membership. Every member must restore the same selected snapshot, with its own name and peer address and a consistent initial-cluster specification. Record the new cluster token, revision-bump decision and compaction requirement. Check addresses against certificate identities. The same bytes and revision policy must be used across members.

Align paths and persistent configuration. A different data directory affects both the static-pod hostPath and kubeadm's stored configuration. If only the live manifest changes, a later upgrade may recreate the old path. Record the expected file ownership, runtime environment and disk mapping in the version-specific procedure.

Validate etcd before API access. The acceptance record should show the expected member count, identities, endpoint health and access using the intended certificates. An unexpected member or failed endpoint is a stop condition. Do not restart clients simply because one member is listening.

Restore clients in a controlled sequence. Confirm API-server endpoints and encryption configuration, then check API readiness and the controllers' state. Client certificates may be expired even if the snapshot is valid; kubeadm certificate-expiration reporting helps separate that problem from database recovery. A readiness response establishes control-plane readiness, not application data integrity.

The following queries provide post-recovery evidence once an authorised procedure has restored the service:

kubectl get --raw='/readyz?verbose'
kubectl get nodes
kubectl get pods -A

Capture their status and output with the snapshot identifier and measured recovery times. The real acceptance decision also needs the persistent-volume and application checks below.

If a machine was replaced

The steps above assume each node still has /etc/kubernetes and its old address. A replacement has neither, so before restoring membership it needs the following:

• The CA and service account files, the static pod manifests, the kubeconfig files and the encryption configuration, from the backup taken alongside the snapshot.
• New etcd server and peer certificates if the address or host name changed. kube-etcd and kube-etcd-peer both require the host name and host IP, so a certificate issued for the old machine will not fit a new address.
• An etcd.yaml and an --initial-cluster value that use the new machines.
• The same container runtime, kubelet and matching etcdutl.

The pages we opened do not walk through this case end to end, so rehearse it if your failure plan depends on it.

When the last snapshot is too old

Kubernetes says the restore file can be a snapshot from a previous backup or from a remaining data directory. When quorum is gone and the last snapshot is old, member/snap/db from a stopped survivor may be newer than every backup. The price is that it may lack data held only in the write-ahead log, and it restores only with --skip-hash-check.

etcd's other route is to create a new cluster from the old data, force a single member to act as leader, then add members one at a time. That is what --force-new-cluster is for, on a survivor, with every other old member confirmed down. Copy the data directory before trying either. Whether recent data is worth that risk is the operator's call.

Reconciling the cluster with reality

The snapshot is older than the real world. After a restore, expect differences between what etcd says and what exists (our reasoning, so check each on your cluster):

• Objects created since the snapshot are gone, while their pods, load balancers or volumes may still exist.
• Objects deleted since the snapshot come back.
• Storage volumes created since may be orphaned, and volumes deleted since may be referenced by restored objects.
• Nodes added since may need to rejoin.
• Credentials rotated since come back in their old form.

Make a list from your change log for the period between the snapshot and the failure, and walk through it. Then re-apply anything held in source control.

Prove it before you need it

A restore drill needs a separate environment with no path to production. The plan should restore the selected snapshot using matching tools, verify keyspace access, then exercise a complete control plane with the corresponding keys and configuration. Use disposable storage and record the expected member identities and endpoints before the drill; no executable restore sequence is supplied here.

The restored keys should list your namespaces. Record the time taken. The drill host now holds your Secrets, so treat it and the test directory accordingly, and wipe the storage when finished.

This shows the snapshot is readable. Only a full control plane rehearsal shows that your runbook works.

Risks and limits

• Never restore with API servers running. Kubernetes states this directly.
• --force-new-cluster is a last resort. The etcd project strongly discourages it, because it panics if members of the previous cluster are still alive.
• Application data is separate. A control plane restore does not bring back persistent volume contents. Protect those by other means.
• Scaling is not a backup. Kubernetes recommends static five-member clusters and no autoscaling of etcd for production. More members tolerate more failures, and a bad delete still replicates to all of them.
• Not covered: external etcd topology in detail, loss of the cluster CA, and restores on managed services.

Sources

Opened 2026-11-10.

• Kubernetes: Operating etcd clusters, PKI certificates, Encrypting data at rest, Secrets good practices
• Kubernetes: Static pods, kubeadm upgrade, kubeadm certificates, kube-apiserver reference, API health endpoints
• etcd: Disaster recovery, Failure modes, Runtime reconfiguration, Maintenance, Clustering, Data model, Install
• etcdutl README, etcd 3.6 changelog
• etcd 3.5 changelog, etcd v3.7 disaster recovery, etcd v3.7.3 release, kubeadm 1.37 constants

This article was originally published on 10 November 2026.