Australian cloud for business software — built and run in Sydney since 2003

NetLeverage
NetLeverage

Who may say a restore test passed, and who may stop it

Technical article
Backup & Recovery

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A restore test passes only when a named person compares the result with a success criterion that was written before the test, and records the comparison. It does not pass by applause, by a green job, or by the most senior person in the room deciding the day has gone on long enough.

Stopping is a different decision from passing. NIST SP 800-84 says the test director, and any member of the test team, should have the ability to terminate the test immediately when a catastrophic disruption is possible, when a person's safety is at stake, or when the security of the organisation or its data is in question. That authority is there to be used. A culture that treats a stop as a failed career, rather than as a required control, will not have the stop when it matters.

Returning a system to normal service is a third decision. SP 800-34 describes it for real reconstitution, not for every sample restore in an isolated network. Mixing the three decisions is how a lab sample gets written up as "production recovered" or how a dangerous test continues because nobody wants to be the person who halted a pass.

The role names below are the names in the US federal guides that were opened. An Australian organisation maps them to its own job titles in the contingency plan. The Information Security Manual does not, in the backup section opened for this article, invent a replacement set of titles. The mapping is your plan's job. This article does not create new statutory roles.

No exercise was run. The examples are hypothetical.

Why it matters

The controls discussed here apply where your organisation has selected or is required to meet them. They are not a blanket legal requirement for every business.

SP 800-34 says a test plan examines selected elements against explicit objectives and success criteria, so effectiveness can be assessed. If the criterion is written afterwards, the assessment is a story. If nobody is assigned to make the comparison, the story belongs to whoever speaks last. CP-4 in SP 800-53 requires the test, a review of the results, and corrective action where needed. Review means a person looks at the result against what was required. Corrective action means a miss is not relabelled as a pass.

SP 800-84 separates running the test from judging it. During the test, the test director monitors. Any team member can terminate for the three conditions above. After the test, the director leads a short debrief and collects notes. Then a member of the design team, or another selected staff member, writes the after-action report that determines how well the tested systems functioned. The report includes scope, objectives, observations, recommendations, and the participant list. Passing is what that report is for. It is not a hallway verdict that never reaches the report.

SP 800-34's reconstitution section is easy to over-apply. After validation testing of a real recovery, contingency plan personnel declare that reconstitution is complete and the system is operating normally. That declaration may sit in a recovery log. Separately, the contingency plan coordinator, with the system owner and the security officers, and with the concurrence of the authorising official, decides whether the system changed enough to need reassessment and reauthorisation. Those sentences are about coming back into service and about whether the change was large. They are not a requirement that the authorising official sign every weekly file-open. Using them for every sample either buries the official in trivia or, more often, means nobody declares anything because the official was never going to attend.

The safety side is not optional colour. SP 800-84's reason for immediate termination includes the security of the data, not only physical danger. A restore test that starts writing into production, or that exposes a copy it should not, is a security question in that sentence. The person who sees it does not need permission from the person who hopes the test will pass.

How it works

Write the criterion, then name the comparer

Before the test, the plan states the criterion in observable terms. SP 800-34 wants that, and it wants quantifiable metrics where the event is a test rather than a discussion. "The restored ledger opens and the control total matches the figure stored with this plan" is a criterion. "Recovery looks good" is not.

Name one role as the comparer for that event. SP 800-84's structure points at the test director for the conduct and the debrief, and at the after-action author for the written determination. Those can be the same person in a small team. They should not be a person who can change the criterion during the event. If the comparer is also the operator, the plan still requires the expected outcome to be the one written earlier. The operator does not get a private, improved definition of success at 16:00.

For a tabletop, the comparer is judging the discussion against the objectives SP 800-84 sets for tabletops: whether the plan's content and the roles held up. A tabletop comparer who writes "restore passed" has changed the event type. The facilitator can recommend. The after-action author records what was shown, which is a conversation, not a retrieved backup.

For a sample restore, the comparer records met or not met against the criterion, plus the backup identity and the clock. A sample can pass its own criterion and still be a narrow pass. The report's first line says it was a sample. CP-9(2) allows samples. It does not allow samples to be renamed as full reconstitution.

What "pass" is allowed to ignore, and what it is not

A pass can sit beside open corrective actions if the criterion was met and the actions are about something outside that criterion. Say so. Example: the file opened from the offsite copy, and the team still has an action to document the key location. The open is not a failure of that criterion. Hiding the action would be. CP-4's corrective action is part of the review, not a stain that forces you to fail a criterion you met.

A pass cannot ignore a miss of its own criterion. If the control total did not match, the result is not met. Partial credit is allowed only if the plan defined partial criteria. Otherwise the comparer records not met and the after-action report carries the recommendation. SP 800-84 says the report determines how well the systems functioned. "Well" is not a synonym for "we tried."

Do not quietly merge a terminated attempt into a later pass. As a local governance rule, preserve the stopped attempt and require an authorised restart with documented safeguards and separate results. SP 800-84 establishes immediate-stop authority; it does not prescribe this particular restart record format. The stopped event's result is "stopped," with the reason: possible catastrophic disruption, safety of a person, or security of the organisation or its data. Those are the three triggers the publication names. Add, in your own plan, any narrower stop you need, such as "isolation to production failed" or "the backup generation is not the one named in the plan." Those narrower stops are editorial additions. They fit under the publication's security-of-data trigger when the alternative is to keep going on the wrong copy or on a path into production. They are still worth writing in local words so a team member does not have to litigate the federal sentence in the moment.

Any member of the test team holds that stop, not only the director. SP 800-84 is explicit. A plan that says "only the manager may halt the test" contradicts the guide you are claiming to follow. Map "test team member" to the people actually on the test, including the person watching the network path, not only the person at the keyboard.

The declaration you use only when service really returns

When the event is a real return to operation, SP 800-34 adds the reconstitution declaration. Contingency plan personnel declare, in a log or equivalent, that reconstitution is complete and the system is operating normally. That declaration comes after validation: data testing that files or databases are complete and current to the last backup you are relying on, and functionality testing that the system can do its work. A declaration without those checks is the sentence the guide does not authorise.

The same section gives a second decision to a named group. The coordinator, in coordination with the system owner, the information system security officer and the senior information security officer, and with the concurrence of the authorising official, determines whether the change is significant enough to need reassessment. The guide's footnote examples of significant change include a new or upgraded hardware platform and a move to a new facility. A restore onto replacement hardware can be that kind of change. The weekly isolated sample usually is not. Put the question in the plan so the official is asked when the footnote's examples apply, and is not used as a rubber stamp on drills.

Deactivation of the plan, in the same section, includes telling users, cleaning up temporary recovery sites, returning media offsite, taking a fresh backup, and filing the after-action report. Keep the declaration of validated service recovery separate from formal plan deactivation. SP 800-34 places a fresh full backup as soon as reasonable after reconstitution and formally deactivates the plan after the close-out activities and documentation are complete. Record any outstanding actions; do not move the post-reconstitution backup ahead of the recovery declaration as if the guide required that order.

Disagreement

If the operator says the criterion was met and the comparer says it was not, the written criterion decides, and the after-action author records the disagreement. SP 800-84's debrief exists so observations are collected while they are fresh, from participants and from data collectors, not only from the operator. Data collectors' notes are inputs to the report. They are not overruled because they are inconvenient.

If a team member stops the test and the director disagrees, the publication has already chosen: the team member has the ability to terminate immediately for the three conditions. Under the local restart rule proposed here, the director can convene a new test later after addressing the cause of the stop. The director cannot treat the stop as invalid because it spoiled the schedule. Write that into the local plan in a sentence people will actually say aloud.

What to choose and why

Choose titles you have, mapped in a single table in the plan: who writes the criterion, who operates, who compares, who may stop, who writes the after-action report, and, for a real return to service, who makes the reconstitution declaration and who must concur on reassessment. SP 800-34 already expects teams to know their goals and the interdependencies between teams. The table is that expectation applied to the verdict.

Choose the comparer to be someone who can read the criterion and the evidence, not necessarily the most senior executive. Senior concurrence is the SP 800-34 reassessment question, used when the system actually changed underneath production. For a sample, a senior signature that does not look at the backup identity adds delay and subtracts nothing from risk.

Choose to let the stop be slightly too sensitive rather than slightly too brave. The cost of a stopped drill is another drill. The cost of a drill that damages production data is the thing SP 800-84 told the test team to prevent. CP-4 still wants you to test. The stop rule is how you test again next quarter.

Choose to record "not met" without requiring the comparer to propose the fix in the same breath. The recommendation can follow in the report. Blocking the verdict on a perfect action plan is how fails remain unofficial.

For a hypothetical finance restore: the plan's criterion is the control total. The operator restores the offsite copy in the isolated network. The comparer, who did not edit the criterion that morning, checks the total and writes "met" or "not met" with the backup's timestamp. A network watcher who sees a route to production appear says stop, and does not need the comparer's permission. Nobody declares "operating normally," because production never moved. That last restraint is the difference between a passed sample and a reconstitution.

Risks and limits

The first risk is a senior override. The authorising official's concurrence in SP 800-34 is about reassessment after significant change. It is not a wildcard to mark a missed criterion as passed. If your local mapping gives an executive a veto, write what the veto can change. It should be able to refuse a return to service. It should not be able to edit the historical result.

The second risk is a stop rule that exists only in the federal PDF. If the people on the test have not been told they may halt it, they do not have the ability SP 800-84 describes, whatever the PDF says. Say it in the briefing SP 800-84 already expects at the start of a larger test.

The third risk is declaring normal operation from an isolated test. An isolated exercise may rehearse full recovery and reconstitution of its test systems. It cannot declare that production service was restored when production was never moved. State which environment the declaration concerns.

The fourth risk is a pass with a hidden generation change. The criterion included a point in time, the copy that opened was older, and the comparer was shown only the application screen. The comparer has to see the backup identity, not only the screen. Otherwise the role is ceremonial.

The fifth risk is importing US federal titles as if they were Australian appointments. Information system security officer and authorising official are terms in SP 800-34. Use them as a checklist of decisions: who owns the system, who owns security, who must agree that a big change needs reassessment. Give those decisions to roles you actually have. Do not leave them blank because the imported title does not exist on your chart.

SP 800-84 is from 2006 and SP 800-34 from 2010. CP-4 in the Rev. 5 control catalogue still cites both, and its own text still requires review and corrective action. Nothing opened here names a vendor's approval workflow. A product button labelled approve is not the role unless your plan says it is.

This article was written with AI assistance. It is source research only. The factual claims have been independently checked against the cited sources. It does not appoint anyone and it does not replace your contingency plan.

Sources