What to Do After an Incident Post-Mortem

After an incident post-mortem, add a replay test to the highest-priority corrective action before anyone closes it. A corrective action can be completed while the condition that caused the incident remains possible. Check whether the proposed change blocks that failure, then name the signal that would show the protection has stopped working.

This is the work after the review. The timeline is agreed, the contributing causes are recorded, and the team has chosen its changes. The next decision is whether the evidence supports closing each action and relying on it during normal operations.

An incident post-mortem records an incident, its impact, contributing causes, response, and follow-up actions intended to reduce recurrence or harm.

Give each corrective action a test and an owner

Start with the action list approved at the review. Assign each action to one person, give it a due date, and link it to the incident evidence. Separate work that restores service from work that prevents the same failure. A temporary restriction can remain necessary while a permanent change is being prepared.

Google's post-mortem guidance includes follow-up actions in the incident record and puts effective prevention among the review's main goals. That gives the action list a clear purpose. It should change how the system behaves when the relevant conditions return.

Rewrite vague actions before assigning them. “Improve testing” does not identify a change or a passing result. An action to add the incident configuration to a regression test does. Record which version will contain that test, who will check its result, and what result would prevent release.

Write the closure condition when the action is assigned, rather than after the work is finished. This keeps the test tied to the incident. It also lets the owner identify missing logs, configuration snapshots, or access before those gaps delay verification. Keep customer data protected when preserving the material needed for a replay.

Keep the distinction between accountability and ownership practical. The action owner organises the change and its evidence. The person accepting closure checks whether that evidence meets the stated condition. For an action with serious consequences, arrange a check by someone other than its implementer.

A delivery date tells the team when work is due; a closure condition tells it what must be demonstrated. Both belong on the same ticket. Neither requires a separate reporting system.

The post-mortem names the correction. What evidence supports closing its action? Start the Walk →

A good review makes follow-up work possible

A careful post-mortem gives the owner material that an ordinary defect report may lack. It identifies the sequence of changes, the affected service, the observed failure, and the response that restored operation. Those details help turn a proposed correction into a testable claim.

Keep that material attached to the action. Someone who joins the work later should be able to see why the test uses a particular configuration. They should also see which details remain uncertain. A missing snapshot is a limitation to address, not permission to replace the incident with an easier test.

Google's guidance on action items warns against vague wording and missing ownership. It recommends a single owner with collaborators and a formal way to track actions. These are useful safeguards against work disappearing between teams. They make implementation visible and give someone responsibility for bringing back the result.

Keep the evidence from the review connected to the evidence used for closure. A change request, test result, deployed version, and monitoring rule should be traceable from the original action. A manager can then check the actual protection rather than accept a summary that everything is complete.

This also makes handover clearer. The operations team needs to know whether the temporary workaround remains active, which change has reached production, and how a recurrence will appear. “Fixed” is too broad if those states differ. Record them separately so the next shift does not remove a workaround prematurely.

A review of the assumptions behind the original decision can explain why the incident occurred. Follow-through asks a different question: what evidence supports relying on the chosen correction now? That is an important part of decision quality, even when the original analysis was thorough.

Would the fix survive the configuration that caused the outage?

Fastly's outage on 8 June 2021 shows why recovery and permanent correction need separate records. In its company account of the outage, Fastly reported that a software deployment begun on 12 May introduced a latent bug. A valid customer configuration change later triggered it, causing 85% of its network to return errors.

Fastly detected the disruption within one minute. It identified the configuration and disabled it; within 49 minutes, 95% of the network was operating normally. The company also reported creating a permanent bug fix and beginning its deployment at 17:25 UTC. Removing the trigger and deploying the correction were distinct actions.

The account announced further examination of why quality assurance and testing had missed the bug, plus ways to improve remediation time. These were stated follow-up commitments. The account does not establish that those reviews were completed or show a replay test proving the correction. It cannot support claims about their final outcomes.

Applied to a similar corrective action, the practical test is specific. Preserve the configuration that activated the defect. Recreate the relevant circumstances in a controlled environment, verify that the old version fails, and run the same test against the corrected version. Do not replay a disruptive incident against live customers.

A passing test matters only if it exercises the condition the correction is meant to handle. If the old version passes too, the test may not reproduce the failure. Check the request path, configuration combination, system state, and order of operations before accepting the result as evidence.

Then name what the test leaves unverified. The production system may have a different dependency version, a larger workload, or another configuration that reaches the same faulty path. Treat those as assumptions behind the fix. Select further checks according to their effect on service and the consequences of being wrong.

This is where assumptions in decision-making become useful after the review. The task is no longer to reconstruct the original judgement. It is to decide whether the correction and its deployment have enough supporting evidence. Close the action against the tested scope, and state the remaining limits explicitly.

From a closed ticket to a watched condition

Before releasing the correction, choose the observation that would reopen the action. A recurrence signal should identify the relevant failure closely enough to prompt a useful response. Record the affected request type, configuration or service, the threshold for intervention, and the person responsible for acting.

Google's monitoring guidance distinguishes symptoms from causes. Use that distinction here. A user-facing error shows harm, while a configuration or internal check may help identify its cause. Keep enough detail to connect an alert to the incident action rather than label every service error a recurrence.

Test the alert as well as the correction. Check that the signal is collected, the rule detects it, and the notification reaches the current owner. Record the immediate response: pause further deployment, apply the safe workaround, or use the agreed rollback procedure. Check that response remains possible after the change.

A canary release evaluates a change on part of a service before wider rollout. If using that approach, inspect the affected group separately. An overall error rate can hide a failure in a small deployment. The canary must receive the relevant configuration and workload to provide useful evidence.

Keep the recurrence signal and response active after the implementation ticket closes. Review them when configuration rules, dependencies, or service ownership change. The Universal Decision-Making Method includes monitoring during implementation because the conditions supporting a decision can change. The action record should identify those conditions in terms the operator can check.

The final handover should therefore include the replay result, deployed scope, remaining limitations, and the signal that reopens the decision. Connect it to the team's existing monitoring practice. A closed action is acceptable when its evidence is sufficient and its owner can respond if that evidence stops holding.

Grant Purdy is the co-author, with Roger Estall, of Deciding (2020), and the architect of the Universal Decision-Making Method.