Messaging backbones before the applications that consume them

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Bring the event log or the queue back to a state that can accept and keep messages before you start the applications that publish or consume them. An application database may also be a system of record; recovery must reconcile it with the retained messages and processing positions. Kafka's introduction describes producers and consumers as fully decoupled: producers do not wait for consumers, and events stay in the topic after they are read, until a retention setting discards them. RabbitMQ's quorum queues confirm a publish only after a majority of members have the message. In both designs, the applications are clients. The backbone is the thing that has to be true first.

This article does not give commands to start a broker, reset a consumer, or delete a queue. Deleting a queue that has lost its majority is a destructive act the RabbitMQ page names, and it was not performed here.

Why it matters

NIST SP 800-34 Rev. 1 sets a resource's recovery time by the impact on the other resources that need it. A payment service, an order service and a notification service can all be "up" and still be useless if the stream they share is down, empty, or split so that each side has a different history. Restoring the applications first feels faster, because those are the things with a business name. It produces clients talking to a backbone that is not ready, and the clients will remember that conversation.

Kafka's introduction calls event streaming the practice of capturing events, storing those streams durably for later retrieval, processing them as they happen or afterwards, and routing them where they need to go. The durable store is the recovery object. The processing can be repeated, because the introduction says events can be read as often as needed and are not deleted when a consumer reads them. That is a gift and a hazard. A consumer that starts again from the beginning will reprocess the retained history. A consumer started against a wrongly recreated topic can encounter missing or incompatible offsets. Whether it fails, replays or skips records depends on its group state, reset policy and the restored log; an empty topic does not by itself prove permanent skipping.

How it works

What Kafka says the cluster is

The Apache Kafka 4.4 introduction describes a distributed system of servers and clients speaking a TCP protocol. Some servers are brokers, the storage layer. Others may run Kafka Connect to import and export streams, including to databases and to other Kafka clusters. The introduction says a cluster is fault-tolerant: if any server fails, the other servers take over its work to ensure continuous operations without any data loss. It then explains the mechanism you actually configure. Every topic can be replicated, including across sites, so that several brokers hold a copy. A common production setting, it says, is a replication factor of three, and replication is per partition.

Those sentences have to be read together. The "without any data loss" line is the introduction's description of a fault-tolerant cluster, not a test result for a topic that was never replicated. A one-copy partition becomes unavailable when its sole broker fails; permanent data loss depends on whether its storage can be recovered. Our recommendation: the recovery gate is not "a broker process is listening". It is "the partitions you care about have their expected data and leaders, and their in-sync replicas, producer acknowledgements and minimum in-sync-replica settings meet the approved durability policy". The introduction does not define the in-sync check. It does say the copies are why a failure can be survived. Do not declare the cluster recovered because one broker booted.

Events have a key, a value, a timestamp and optional headers. Producers write them. Consumers read them. Producers and consumers do not know each other, and producers do not wait for consumers. That decoupling is why you can restore the brokers without the consumers, and why restoring the consumers does not recreate the log. Topics are multi-producer and multi-subscriber. With a consistent key-based partitioning configuration and stable partition count, events with the same key go to the same partition, and a consumer of that partition reads them in the order they were written. Order is not a property of the whole topic across all partitions. A recovery that rebuilds a topic with a different partition count, or that lets two producers write the same key to different places because the topic was recreated by hand, can change key routing and application-level ordering across the transition, even though each individual partition retains its own order. Recreate topics from the recorded configuration, not from memory.

The introduction says retention is configured per topic. Inspect the actual cleanup policy and retention limits: time-based deletion, size limits and compaction do not preserve the same history. A long recovery can outlast the required backlog even while brokers are healthy; expiry is not a universal exact-time deletion promise. Our recommendation: before the outage, know the retention of the topics that are the system of record, and know whether it is longer than the recovery you can actually perform. During the recovery, do not shorten retention to "save disk" on a half-built cluster. You would be deleting the history the consumers still need.

Kafka Connect is how streams meet other systems, including databases. The introduction gives a database connector that captures table changes as an example, and it says you usually use an existing connector rather than writing one. A connector is another client. It should not be the first process you start, and it should not be the last thing you think about. A connector that writes into a database you have not restored, or that reads a database and publishes a second copy of history into a topic that already has it, doubles the world. Leave connectors until both the topic and the external system are the copies you mean.

The introduction also says clients exist for many languages, and that the project ships a Streams library for processing. Those are applications. They belong after the log.

What a quorum queue will and will not promise

RabbitMQ's quorum-queue page, marked version 4.4, defines this queue type as a durable, replicated queue based on the Raft consensus algorithm, and it says to treat it as the default when you need a replicated, highly available queue. Classic mirrored queues were removed in RabbitMQ 4.0. A recovery document that still says "fail over the mirrored queue" is describing a feature the current page says is gone. Streams are the other replicated structure, for cases that need replication and repeatable reads, such as fan-out. This article stays with quorum queues, which the page aims at long-lived queues where losing a message would make the system wrong. Orders are the example it gives. Stock tickers and chatty reply queues are the examples it says benefit less.

A quorum requires a strict majority: floor(N/2)+1 members. Applied to the queue, the majority of members, leader and followers, agree on the state of the queue and its contents. A leader is elected when the queue is formed and again if the leader becomes unavailable. A follower can be elected and resume. Followers that return replicate from where they left off. There is no separate catch-up mode that takes the leader offline.

The safety rule is the one to put in the runbook in plain language. Publisher confirms are how a client talks to that consensus. A confirm is issued only once the message has been replicated to a quorum of members. A message that was confirmed should not be lost as long as at least a majority of the nodes hosting that queue are not permanently unavailable. Messages that were not confirmed can be lost in flight or in an operating-system buffer. The page says quorum queues favour consistency over availability, and that they cannot provide safety guarantees for unconfirmed messages.

Availability follows the same majority. A queue with three members tolerates one node down. Five members tolerate two. If a quorum of nodes cannot be recovered, the page's example is two of three permanently lost, the queue is permanently unavailable and will need to be force-deleted and recreated. That is not a heal. It is a new, empty queue. Our recommendation: do not force-delete because a node is merely stopped. Stopped is the case the election is for. Force-delete is for a majority that is gone for good, and everyone who produced or consumed that queue has to be told the history ended.

The page's table is blunt about small clusters. One node tolerates no failure. Two nodes tolerate no failure and are not tolerant of a partition. Fewer than three nodes do not get the full guarantees. An even number of nodes does not benefit from spreading members across all of them; constrain the queue to a smaller odd size. Performance, it says, tails off above five, and it does not recommend more than seven. The default size is three. A recovery site with two RabbitMQ nodes is not a highly available quorum queue, whatever the diagram from last year said.

The order

Our recommendation:

1. Decide whether the backbone still has its data. For Kafka, that means the brokers that hold the replicated partitions, not a fresh cluster with the same name. For a quorum queue, that means a majority of the members that had the queue, not two new nodes. A new cluster is a new log. Pointing old consumers at it does not replay what was lost.
2. For RabbitMQ quorum queues, restore a majority of that queue’s members before accepting publishes. RabbitMQ will not confirm a message until a quorum has it. Publishing into a minority fills the client's retry logic and, for anything not confirmed, the page says the message can be lost. Do not apply this RabbitMQ majority rule to Kafka partition writes. Kafka requires its own checks of leadership, in-sync replicas, acknowledgement policy and minimum replicas; controller quorum is a separate concern.
3. Then start consumers, and know where they will read from. Kafka does not delete events on read, so a consumer that begins at the start of the retained log will process history again. A consumer that begins at the end will skip anything it had not acknowledged before the outage. Normal resumption may instead use committed group offsets. The 4.4 introduction does not specify offset restoration, commit timing or reset behaviour. Our recommendation: the application owner verifies committed offsets, any reset decision and the matching downstream processing state, in writing, before the process starts. Reprocessing orders is a business decision. Skipping them is also a business decision. Neither should be the default of a library nobody checked.
4. Use manual acknowledgements on quorum queues, which is what the page tells consumers to do so that a message that was not processed returns to the queue. Acknowledge only after the required processing is durable. Redelivery can duplicate side effects, so idempotency or explicit reconciliation is still necessary; manual acknowledgement alone does not provide exactly-once business processing.
5. Start producers after confirms or equivalent durability are on. A producer restored from an old configuration that does not wait for a confirm will take the page's "no safety guarantee" path.
6. Connectors and bridges last, once both the backbone and the other system are the right copies. Kafka Connect is an integration client, not the storage layer.
7. Do not shorten retention, and do not force-delete, to make a dashboard green. Retention discards events. Force-delete discards the queue. Both are data-loss actions wearing an operations hat.

Independent applications that only share the backbone can start together once the backbone gate has passed. They may start in a paused or retrying state beforehand, provided they cannot consume, publish or reset positions unsafely. NIST's priority among business processes still decides which consumer gets the people. It does not justify starting a consumer against a topic that is empty because the brokers were never the ones that had the data.

What to choose and why

These are our recommendations.

Risks and limits

• The Kafka sentences are from the 4.4 introduction, not from the design chapter. The page itself points to the design section for full detail. Replication factor, leader election and consumer-offset settings are not fully specified in the introduction. Where this article goes beyond the introduction, it says so.
• "Without any data loss" is the introduction's description of failover in a fault-tolerant cluster. It is not a measured result, and it is not a promise for an unreplicated topic. A common setting of three copies is also from that page.
• RabbitMQ's guarantee is for confirmed messages, and only while a majority is not permanently lost. Unconfirmed messages can disappear. A minority cannot serve. Force-delete is permanent.
• Classic mirrored queues are gone as of RabbitMQ 4.0, on this page. Do not recover toward a feature the vendor removed.
• No broker was started. Partition counts, disk sizing and client offset settings were not tested. The replay-versus-resume choice is a decision the introduction makes possible, because events survive being read. It is not a documented default of any client in the pages opened here.
• NIST does not mention Kafka or RabbitMQ. It is used only for the rule that a shared resource is timed by the systems that depend on it.

Research and verification

Drafted with AI assistance on 11 November 2026 and independently source-reviewed on 13 November 2026. No article commands or recovery operations were executed. This is educational source research, not a lab-validated production procedure.

Sources

Independently reopened 13 November 2026.

• Apache Kafka 4.4, Introduction. Brokers, replication, producers and consumers, retention, the fact that events are not deleted when read.
• RabbitMQ, Quorum queues, documentation version 4.4. Majority, publisher confirms, data safety, what happens when a quorum cannot be recovered. Classic queue mirroring removed in 4.0.
• NIST SP 800-34 Rev. 1, May 2010: PDF. Recovery time as impact on dependent resources.

This article was originally published on 13 November 2026.