What to Do After a Disaster Recovery Test

After a disaster recovery test, write the loss scenario beside the measured recovery time before approving recovery readiness.

On 10 March 2021, a fire at OVHcloud’s Strasbourg site required a power cut across all four datacentres. Its recovery shows why a test result needs a clear statement of what remains available when something fails.

A disaster recovery test checks whether planned procedures can restore systems and data under specified disruption conditions.

OVHcloud’s fire interrupted recovery across the Strasbourg site

The fire began at SBG2. OVHcloud’s initial update reported that SBG2 was largely destroyed and SBG1 partly damaged. SBG3 and SBG4 were undamaged but switched off. Physical separation into buildings did not keep every building operating while firefighters worked.

The company’s Strasbourg incident chronology describes inspections, electrical and network work, soot removal, and replacement capacity at other sites. Servers returned through a controlled sequence of cleaning, inspection, installation and restart. Smoke detected on 19 March required another power cut at SBG1 and SBG4, delaying the resumption of services.

Those facts show several conditions affecting restoration. They do not show that OVHcloud had passed a particular recovery test before the fire. They also do not establish what any customer had tested. The incident supports examining recovery dependencies; it does not prove that a documented test failed.

The practical question after a test is therefore precise: would the measured result still apply if the resources used during that test were unavailable? A restore performed with the usual administrator account, network and backup store can demonstrate a real capability. The same result needs further evidence before it supports a claim about losing those resources together.

Take the completed test report to the service owner. Locate its stated starting conditions and compare them with the proposed readiness statement. Did the test remove the whole site, one server, or only a file? Did it restore an application from backup, switch to a running replica, or demonstrate that a recovery script completed? Each result supports a different claim.

This is a hypothetical comparison, not a description of OVHcloud’s test arrangements. If a restore starts with an accessible backup already selected, the result includes neither finding that backup nor regaining access to it. If someone manually prepares the replacement environment beforehand, the timed result excludes that preparation. Those exclusions can be appropriate, provided the approval names them.

State what the test depended on before using its elapsed time as evidence of readiness. Record the location of the recovered data, the account used to retrieve it, and the infrastructure used to start the service. Put any resources supplied by the test team beside the result. A reader should not have to reconstruct these conditions from separate technical logs.

Shared resources matter even when the recovery copy sits elsewhere. Ask whether the replacement environment needs identity services, encryption keys, deployment tools or configuration held in the environment being lost. The technical team can trace that route. The service owner must decide whether the evidence supports the recovery claim being accepted.

The test may remain useful after this review. It can support approval for a server failure while site-loss recovery remains unresolved. That is a narrower decision with identifiable evidence. It gives the person responsible for deciding under pressure a clear limit to work with during an incident.

For leadership under pressure, this distinction also shapes what can be promised. Describe the service and conditions covered by the evidence. Leave an untested recovery route open for a separate decision.

Your recovery time was measured under specific conditions. Which of those conditions would disappear in the loss scenario you are approving? Start the Walk →

What does the recovery test actually justify?

A recovery test deserves to influence the decision. It can expose an unreadable backup, a missing configuration or an incorrect restore sequence. It also gives operators direct experience of the procedure. NIST’s guidance on tests and exercises distinguishes testing systems from exercising plans and training people. These activities answer related questions, but their evidence differs.

Preserve the actual result before making corrections. Keep the timing record, restored data version, errors and manual interventions. A successful second attempt should not erase the failed first attempt. The service owner needs to see what changed and what still depends on someone recognising and correcting a problem.

A passed test justifies the recovery capability demonstrated under its recorded conditions. It does not justify every condition that the readiness statement might cover. This is the point to examine assumptions in the decision, including the assumption that a working resource during the test will survive the intended loss scenario.

The chain below is an illustrative technical dependency, not an account of the Strasbourg incident. An accessible backup may still depend on an account that cannot be used after the site is lost.

The recovery time objective is a target for restoring the required capability. The recovery point objective specifies the point in time to which data must be recovered. Check the observed result against both. A quickly restored system with data older than the accepted limit does not demonstrate the proposed recovery requirement.

Then check what the business could actually do. Agree on a small, observable task that demonstrates the required service. It might be retrieving a complete record or processing an authorised transaction in a safe test environment. Define the required output, data integrity and access checks before declaring success.

Accept readiness when the evidence supports the service requirement for the named scenario. The result may support proceeding, approving a limited scope, or delaying acceptance for a targeted test. The relevant question is whether the information is sufficient for this decision. Another broad exercise is unnecessary if one specific restore check can settle the unresolved condition.

Set the evidence required before accepting recovery readiness

Use the test report to make the next decision explicit. A disaster recovery plan describes restoration arrangements. NIST’s definition of a disaster recovery plan includes recovery following a major failure and relocation to an alternate site. The approval should say which part of that capability has enough supporting evidence.

Set the acceptance conditions with the service owner. Specify the loss scenario, required service, tolerated data loss and recovery target. Name the dependencies assumed to remain available. Choose the evidence threshold before accepting the result, so the threshold does not change merely to fit a passing test.

The following comparison is illustrative. It shows two ways to use the same file-restore result, not a claimed intervention or outcome at OVHcloud.

Record an unresolved condition as a decision, with an owner and a next action. If backup access depends on the lost environment, arrange a controlled restore without that access route. If replacement capacity was supplied in advance, establish whether it can be obtained within the allowed time. Select the next test because it addresses the uncertain condition that matters to acceptance.

NIST’s contingency planning guide calls for after-action reports, corrective actions and plan maintenance. Keep those records alongside the acceptance decision. Completing an action in a tracker should point to evidence of the changed capability. It should also identify the conditions covered by that evidence.

Use the five steps of the Universal Decision-Making Method to reach and maintain that decision:

1. Frame. State the service that must recover, the loss scenario and the duration of the readiness approval.
2. Tentative Elements. Propose the recovery route and the scope of acceptance, including any limits on service or data.
3. Assumptions. Name the resources that must remain usable, especially backup access, keys, identity services and replacement capacity.
4. Sufficient Certainty. Compare the restore evidence with those conditions. Proceed, narrow the approval or obtain the missing evidence.
5. Implement+Monitor. Assign the unresolved work and specify which changes require the owner to revisit acceptance.

Keep the acceptance record short enough to use during an incident. The review after a cyber tabletop exercise checks whether changed response handoffs require another exercise. Include the approved recovery route, the evidence location and the person who can authorise a change. Record the unresolved scope in the same place, so someone activating recovery can see what has been demonstrated.

Those changes should be concrete: a new backup location, altered identity arrangements, changed encryption keys or a larger restore volume. Link each to the evidence it could invalidate. Monitoring then has a defined purpose: identify when the accepted recovery conditions no longer hold.

---

Grant Purdy is the co-author, with Roger Estall, of Deciding (2020), and the architect of the Universal Decision-Making Method.