Observability has to return before recovery decisions are trusted

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

Stand up a way to see the estate before you trust anyone who says a gate has passed. Restoring last month's metrics database is not that way. It is a history book. The thing you need during the recovery is a live path: something is being measured or recorded now, it is leaving the system you just started, and a person who is allowed to decide can read it.

NIST SP 800-34 Rev. 1 tells contingency planners to install monitoring so that a failing node or connection raises an alert, often before users and other nodes notice. The same guide says reconstitution is not a feeling. It includes validation of recovered data and functionality, followed by a documented declaration; concurrent processing is optional, as the high-impact template explicitly states. You cannot make that declaration from a silent platform. A recorded login test can supply valid functional evidence, but by itself may not reveal component failures or explain what happened between tests.

This is a planning article. It does not install a collector, write an alert rule, or tell you how to repair a dashboard product.

Why it matters

Every earlier gate in an estate recovery is a claim. The name resolves. The address was handed out. The volume is the one you masked. The secrets service is unsealed. A claim without a signal is a conversation. Conversations drift under fatigue, and two teams will report the same service as up and down because they looked at different things.

OpenTelemetry's signals page states the purpose directly. OpenTelemetry exists to collect, process and export signals. A signal is a system output that describes the activity of the operating system and the applications on a platform. It might be a measurement at a point in time, such as memory use, or an event you want to trace through a distributed system. Traces are the path of a request. Metrics are measurements captured at runtime. Logs are recordings of events. If nothing is collecting, processing and exporting, those outputs stay on the machine that produced them, or they are not produced at all. The recovery room does not have them.

Prometheus makes the same split in the language of one monitoring toolkit. It collects and stores metrics as time series, with the timestamp at which the sample was recorded. The server scrapes targets over HTTP, stores the samples locally, and runs rules that can generate alerts. A separate alertmanager handles the alerts. When a server’s only ingestion path is scraping and it stops, that path stores no new samples. Other configured ingestion paths must be assessed separately. The series already on disk keep the timestamps of the old samples. They describe the platform as it was, not the platform you are rebuilding.

How it works

Three different "monitoring" objects

People say "restore monitoring" and mean three things. They fail independently.

Live collection. Something is observing the systems that are up now. For Prometheus, that is a server scraping targets and storing new samples. For OpenTelemetry, that is a pipeline collecting signals and exporting them. NIST's monitoring software, in the telecommunications section, is this object: it notices a node or a link that is not responding, and it warns the administrator before the users do.

Historical telemetry. The time series and logs you had before the outage. Prometheus stores samples with the timestamp at which they were recorded, so a restored disk is a record of those past timestamps. It is useful for the incident timeline. It does not answer whether the server you just started is sealed, unreachable, or serving errors. OpenTelemetry's definition of a log as a recording of an event is the same distinction. A restored archive is a pile of recordings. It is not a recorder.

Notification. Prometheus separates the server that evaluates rules from the alertmanager that handles alerts. Scrapes can be succeeding while nobody is paged, because the component that delivers the alert is down, misrouted, or still pointing at a directory you have not recovered. NIST's value, a warning before users notice, requires this third object as well as the first. Our recommendation: do not report "monitoring is back" until you can say which of the three you mean, and until a test signal has travelled the live path into the place the recovery lead actually watches.

What a live path depends on

Prometheus does not rely on distributed storage. A single server node is autonomous. That is the feature that makes a bootstrap realistic. You do not need the entire historical cluster, or a quorum of monitoring servers, to scrape a handful of targets and answer a query. Collection is a pull over HTTP. The server has to be able to reach the target. Targets come from service discovery or from static configuration.

Those two sentences are the sequencing trap.

Pull means the recovered application does not announce itself. If your only Prometheus is still on the failed site, or cannot route to the recovery network, the application can be healthy and invisible. Push exists, through an intermediary gateway, for short-lived jobs. It is not the default, and a gateway you have not recovered is another silent hop.

Service discovery usually depends on the platform you are still rebuilding: a name, an inventory, an orchestrator. Static targets avoid discovery-service dependencies, although hostnames, TLS and authentication may still require DNS, time or credentials. Our recommendation: the bootstrap set is a static list of the few targets that prove the gates you care about, scraped by one autonomous Prometheus, or exported by one collector that does not itself need the full platform. Service discovery is a later improvement, after the discovery system has passed its own gate. Restoring a configuration that only contains discovered targets, into a network where discovery is dead, produces an empty live view and a green "server is up".

OpenTelemetry's pipeline has the same shape even though the project is not a storage server. Collect, then process, then export. A collector that cannot export, because the backend is sealed, full, or unnamed, is not a live path. Signals may be queued, retried or dropped according to the configuration; until delivery is verified, the recovery lead has no confirmed live view. A backend that is up and receiving nothing, because nothing is collecting, is a blank dashboard. Either failure looks like "we restored observability" if you only check that a process is running.

What to bring back, and when

NIST's reconstitution steps are the consumer of this path. Functionality testing is supposed to show that the system is ready to return to normal. Data testing is supposed to show that what you restored matches the last usable backup. Concurrent processing, where you use it, is supposed to run the recovered system until you have assurance. None of those is a one-off glance. They are easier to lie about, including to yourself, when the only record is a ticket comment.

Our recommendation, not a sentence in any of the three documents:

1. Before the first shared service is declared ready, have an out-of-band view that does not depend on that service. A laptop, a serial console, a status page the product itself serves when it is sealed. Vault, for example, can report status before it can decrypt. Use the product's own status for the bootstrap. Do not wait for a dashboard that needs the secrets, the directory and the name service you have not started.
2. As soon as addresses and names for a small set of hosts work, start one live collector with static targets. The targets are the gates, not the whole catalogue: the name resolver you intend to use, the secrets service, the storage path you will mount, and the first application. A fresh scrape, exported metric or log line visible outside the host proves that specific transport path. It does not prove the service is healthy; the gate also needs a relevant functional check and an explicit freshness limit. A dashboard with two hundred panels is not.
3. Establish suitable observation before declaring applications ready. When you connect an application to the rest of the estate, you already have a place where its errors will appear. NIST's point about hearing from the monitor before you hear from the users is this step. An application may need to start before it can emit useful telemetry. During bootstrap, recorded local checks can supply evidence until the collection path is available.
4. Restore historical telemetry in parallel if you have the people, and do not block the live path on it. The old series answer "what was happening before". They are often large, and they often sit on the same shared storage you are still being careful with. A recovery lead who cannot see current errors because the team is restoring six months of metrics has the dependency backwards.
5. Turn notification on for the bootstrap set before you call the estate stable. Prometheus will not page anyone until alertmanager, or whatever you use in its place, is delivering. Send one test alert to the channel the recovery lead watches. If that channel is the corporate mail system, and mail is down, the test has told you to pick another channel. The test is the gate.

The historical store still matters for the incident. A log recording, in OpenTelemetry's terms, is how you later show what happened. Keep it. Just do not confuse a successful restore of that store with the ability to see the next error.

What "passed" looks like

Write the gate in observables, not in product names. For each layer you start, name one signal you will see if it is working and one you will see if it is not. Examples, not measurements:

• A secrets service reports sealed or unsealed. "Process running" is the wrong signal.
• A name resolver answers a query for a name you chose, from the resolver the next host will use. A dashboard that shows the resolver process using memory is the wrong signal.
• An application trace, in OpenTelemetry's definition the path of a request, either completes or breaks at a named hop. A successful login proves the tested user journey; a trace can help explain its internal path. Neither proves untested journeys or future behaviour, and sampled traces may omit requests.

Prometheus's own overview uses a slow web application as the reason metrics exist: request count and latency tell you whether load is the cause. During recovery the question is smaller. Did the request succeed, and did the error rate move when you connected the next system? You need the series that answer that, with timestamps from now, not a restored series whose timestamps stop at the outage.

Alert rules that fired on the old estate may be wrong on the recovery estate. Thresholds assumed a full fleet. A rule that pages because three of five nodes are down will page continuously while you are still on purpose at two nodes, and people will mute it. Our recommendation: for the duration of the recovery, alerts are few, they match the gates, and a human owns each one. Keep critical security and data-integrity alerts active, and document any temporary suppression with an owner and expiry. Revalidate old rules against the recovered topology before restoring them; do not suppress the whole rule set until a blanket all-clear.

What to choose and why

These are our recommendations.

Risks and limits

• The order is ours. NIST says monitor so you hear about failure before users do, and it defines reconstitution as testing. It does not say "start Prometheus before the application". OpenTelemetry defines signals and a collect-process-export purpose. It does not publish a disaster-recovery sequence. Prometheus describes a pull model, local storage and a separate alertmanager. The sequence is how we apply those facts.
• A restored time series is not live. The timestamp Prometheus stores is when the sample was recorded. Old timestamps do not describe the recovery.
• Pull and discovery hide outages. An empty target list looks like a quiet, healthy system. Static bootstrap targets are how you avoid that reading.
• One collector is a single place to be wrong. Prometheus notes that a server node is autonomous, which is also a single copy. Lose it and you are blind again. The bootstrap is allowed to be small. It is not allowed to be unwatched. A second person should be able to open the same query.
• Nothing here was installed or scraped. No claim is made about how fast any collector exports, how much disk a restore needs, or which dashboard product you should buy.
• Not covered: tracing back-ends, log-shipping protocols, and the security of who may query the bootstrap server. A monitoring server that accepts everyone's metrics and has no access control is a new system on the recovery network. Treat it as one.

Research and verification

Drafted with AI assistance on 11 November 2026 and independently source-reviewed on 13 November 2026. No article commands or recovery operations were executed. This is educational source research, not a lab-validated production procedure.

Sources

Independently reopened 13 November 2026.

• NIST Special Publication 800-34 Revision 1, May 2010: PDF. Monitoring software on telecommunications paths; reconstitution as validation and a documented declaration.
• OpenTelemetry, Signals. Collect, process and export; traces, metrics and logs.
• Prometheus, Overview. Time series and timestamps, pull over HTTP, autonomous single server, alertmanager, service discovery or static configuration.

This article was originally published on 13 November 2026.