About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.
Ask how a Kafka cluster is backed up and the answer is often it's replicated. That answer covers some failures well and others not at all. The cited Apache Kafka 4.4 documentation explains distinct protection mechanisms rather than establishing a complete backup design. It describes is in-cluster replication, retention that removes data on purpose, and MirrorMaker 2, a tool that copies topics between clusters.
This guide sets out what each of those protects against, where the gaps are, and how to decide what a backup should mean for your own event streams.
Why it matters
Kafka often sits between systems: orders flowing to billing, sensor readings flowing to analysis, database changes flowing to search. When it fails, the question is not only whether it can be restarted, but which events were lost and whether every consumer can carry on from the right place.
Those questions have different answers depending on whether the failure was one broker, a whole site, a bad configuration change, or someone deleting a topic.
How it works
What replication inside the cluster guarantees
Kafka copies each partition to several brokers. The design documentation states the guarantee precisely: "The guarantee that Kafka offers is that a committed message will not be lost, as long as there is at least one in sync replica alive, at all times."
The design documentation associates commitment with the message reaching the in-sync replicas and the required minimum ISR being satisfied. It describes replicas applying the message to their logs; this is not a guarantee that all copies have synchronously reached stable media. That set can shrink. The documentation gives an example: "if a topic is configured with only two replicas and one fails (i.e., only one in sync replica remains), then writes that specify acks=all will succeed. However, these writes could be lost if the remaining replica also fails."
Two topic settings let you favour durability over availability:
- Disable unclean leader election. The documentation says that then, "if all replicas become unavailable, then the partition will remain unavailable until the most recent leader becomes available again."
- Set a minimum in-sync replica count (
min.insync.replicas). The required ISR size must meet the configured minimum, so a minimum of two can be satisfied by two in-sync replicas. This "only takes effect if the producer uses acks=all".
The broker setting unclean.leader.election.enable is described as allowing replicas outside the in-sync set "to be elected as leader as a last resort, even though doing so may result in data loss". The design page says that since version 0.11.0.0, Kafka by default favours "waiting for a consistent replica."
The design page also explains why Kafka does not wait for every write to reach the disk: requiring "fsync on every write" can "reduce performance by two to three orders of magnitude." Instead, a replica that crashed "must fully re-sync again even if it lost unflushed data in its crash."
Our reading: replication protects against losing brokers, provided enough in-sync copies survive. It does nothing for mistakes. A topic deletion, a bad batch of records or a shortened retention setting is applied to every replica.
Retention deletes data by design
Kafka is not an archive unless you configure it to be one. The broker setting log.retention.hours is "The number of hours to keep a log file before deleting it", with a default of 168, which is seven days.
Compacted topics behave differently. The design page says log compaction "ensures that Kafka will always retain at least the last known value for each message key". Delete markers are only kept for the topic's delete.retention.ms ("the default is 24 hours"). After that, the page says, "it is possible for a consumer to miss delete markers if it lags by more than delete.retention.ms".
Our reading: before choosing a backup method, decide what you are protecting. For many topics, the honest answer is the last few days of events, so consumers can catch up, which retention and replication may already cover. For others, such as compacted topics that hold the current state of something, losing the topic means losing that state.
What MirrorMaker 2 copies
The geo-replication documentation describes MirrorMaker 2 as "a tool to replicate data between different Kafka environments in a streaming manner." It lists disaster recovery among the common uses. It notes that "This inter-cluster replication is different from" the replication inside a cluster, "which replicates data within the same Kafka cluster."
MirrorMaker 2 can replicate "topics, topic configurations, consumer groups and their offsets, and ACLs". It is built on Kafka Connect. Flows are one-way unless you enable both directions: "if you need two-way (bidirectional) replication, you must enable flows in both directions."
By default, replicated topics are renamed with the source cluster's name in front. The documentation's example is a topic foo in cluster us-west that appears as us-west.foo in the target cluster. Applications that fail over to the copy must therefore use the new names, or you must configure a different naming policy.
The Datacenters page recommends "a local Kafka cluster in each datacenter" with mirroring between them. It warns that it is "generally not advisable to run a single Kafka cluster that spans multiple datacenters over a high-latency link."
Consumer offsets after a failover
Moving consumers to another cluster is often harder than moving data. Offsets on the copy do not match the source one for one, so MirrorMaker 2 records how they correspond.
The MirrorMaker configuration reference shows the defaults:
emit.checkpoints.enabled("Whether to replicate consumer offsets to target cluster") defaults to true, every 60 seconds.sync.group.offsets.enabled("Whether to periodically write the translated offsets to __consumer_offsets topic in target cluster, as long as no active consumers in that group are connected to the target cluster") defaults to false.
Our reading: by default, the target cluster receives offset checkpoints, but consumer groups there are not automatically positioned. Whoever runs the failover must either enable that setting in advance or use the checkpoints to position consumers. Either way, expect some events to be processed twice, because checkpoints are taken at intervals. Design consumers to tolerate repeats.
The basic operations page describes the consumer group tool's --reset-offsets option. It can reset to a time with --to-datetime, and its default execution option is --dry-run, "to display which offsets to reset." It asks you to "first make sure that the consumer instances are inactive."
What MirrorMaker 2 also copies that you might not want
MirrorMaker 2 keeps the target in step with the source. sync.topic.configs.enabled ("Whether to periodically configure remote topics to match their corresponding upstream topics") defaults to true, every 600 seconds. sync.topic.acls.enabled behaves the same way for access rules.
Some settings are excluded by default. The config.properties.exclude list includes unclean.leader.election.enable and min.insync.replicas, among others. Retention settings are not on that list.
Our reading: if someone shortens a topic's retention on the source, the copy may adopt the shorter retention at the next sync and delete its older data too. Bad records, once written, are copied like any others. A mirror protects against losing a site, not against mistakes made on the source. Consider adding retention settings to the exclusion list on a copy intended for recovery, and test the effect.
The geo-replication page also warns about configuration conflicts. MirrorMaker processes "share configuration via their target Kafka clusters", and if two processes are configured differently for the same target, "either the topic foo or the topic bar is replicated, but not both."
There is also a small documentation inconsistency. The geo-replication page gives the default topics.exclude as .*[\-\.]internal,.*\.replica, __.*, while the configuration reference lists mm2.*\.internal,.*\.replica,__.*. Check the behaviour of the release you run.
KRaft metadata: the part to protect most carefully
In current Kafka, cluster metadata is held by a quorum of controllers (KRaft). The KRaft documentation says: "A majority of the controllers must be alive in order to maintain availability. With 3 controllers, the cluster can tolerate 1 controller failure; with 5 controllers, the cluster can tolerate 2 controller failures."
Storage must be formatted with a cluster ID before use. The documentation explains why automatic formatting was dropped: "auto-formatting can sometimes obscure an error condition. This is particularly important for the metadata log maintained by the controller and broker servers. If a majority of the controllers were able to start with an empty log directory, a leader might be able to be elected with missing committed data."
Our reading: this is a warning about recovery as much as provisioning. Restoring controllers from old server images, or re-formatting a majority of them, risks exactly the condition the documentation describes. Our recommendations: keep controllers on separate hosts and failure domains, record the cluster ID with your recovery documents, and never re-format controller storage as a fix without understanding which committed metadata would be lost.
Tiered storage is not a backup
Kafka's tiered storage moves "completed log segments" to "external storage systems, such as HDFS or S3". When enabled, local.retention.ms and local.retention.bytes control how long data stays on brokers before it is eligible for local deletion after remote copying. Offload is not a delayed archival backup created only when local retention expires. "If unset, The value in retention.ms and retention.bytes will be used."
Our reading: the remote copy is still governed by the topic's retention and is managed by Kafka, so deleting a topic or shortening retention applies to it too. It is part of the live topic storage lifecycle, not an independent retained backup. The documentation also lists limitations, including "No support for compacted topics", and notes that "Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation."
Matching protection to failure
Our reading of the documentation, set out by failure:
| Failure | Covered by in-cluster replication? | Covered by a MirrorMaker 2 copy? | Needs something else? |
|---|---|---|---|
| One broker lost | If sufficient in-sync copies and controllers survive | Usually unnecessary for this event | Repair capacity and confirm application progress |
| Whole site lost | No | Yes, up to the replication lag | Consumer failover runbook |
| Topic deleted by mistake | No | Not documented; test it | An independent copy |
| Bad records written | No | No, they are copied too | An independent copy, or reprocessing from the original source |
| Retention shortened by mistake | No | Possibly not, if retention is synced | Exclude retention from sync; an independent copy |
| Controller majority lost | No | The mirror's own metadata survives | Rebuild plan and configuration records |
The MirrorMaker 2 documentation does not say how topic deletions on the source are handled. That row therefore needs a test on non-production clusters before you rely on it.
What to choose and why
These are our recommendations, drawn from the documentation above.
- Classify topics. Separate short-lived event streams, which replication and retention may already cover, from topics that hold state or records you must keep.
- Set durability for important topics. Use a replication factor of at least three,
min.insync.replicasabove one,acks=allfor producers, and unclean leader election left disabled. - Mirror to a second cluster for site loss, with offset checkpoints and a written failover runbook for consumers.
- Protect the mirror from source mistakes. Consider excluding retention from configuration sync, and limit who can delete topics on the target.
- Keep a copy the cluster cannot alter for records you must retain. A sink connector writing to object storage that has its own retention lock is one approach. Kafka's own documentation does not prescribe one.
- Back up what Kafka does not hold for you. That includes broker and controller configuration files, the cluster ID, topic definitions, ACLs, MirrorMaker and connector configuration, and any schema registry you run.
- Rehearse. Fail consumers over to the mirror, measure how many events are repeated, and practise repositioning offsets with the dry-run option first.
Risks and limits
- Replication is not a backup: deletions and bad data reach every replica.
- Retention deletes data on schedule; a mirror can inherit shortened retention.
- Consumer offsets do not move by themselves under default settings.
- Renamed topics on the mirror affect failover.
- Controller metadata loss cannot be repaired by re-formatting.
- Tiered storage follows topic retention.
- Exactly-once mirroring does not prove exactly-once business processing across a failover and needs specific settings; the documentation describes it for dedicated MirrorMaker clusters "as of version 3.5.0".
- Nothing here was run. Behaviour is quoted from the documentation for planning; test on non-production clusters.
Sources
- Apache Kafka 4.4 documentation, Design: https://kafka.apache.org/44/design/design/
- Apache Kafka 4.4 documentation, Geo-Replication (Cross-Cluster Data Mirroring): https://kafka.apache.org/44/operations/geo-replication-cross-cluster-data-mirroring/
- Apache Kafka 4.4 documentation, Datacenters: https://kafka.apache.org/44/operations/datacenters/
- Apache Kafka 4.4 documentation, KRaft: https://kafka.apache.org/44/operations/kraft/
- Apache Kafka 4.4 documentation, Tiered Storage: https://kafka.apache.org/44/operations/tiered-storage/
- Apache Kafka 4.4 documentation, Basic Kafka Operations: https://kafka.apache.org/44/operations/basic-kafka-operations/
- Apache Kafka 4.4 documentation, Broker Configs: https://kafka.apache.org/44/configuration/broker-configs/
- Apache Kafka 4.4 documentation, MirrorMaker Configs: https://kafka.apache.org/44/configuration/mirrormaker-configs/
This article was originally published on .