What to Do After a Crisis Management Exercise

After a crisis management exercise, the step before the improvement plan is closed is to ask what the scenario was never allowed to test.

In July 2004, more than 300 participants spent a week in Baton Rouge responding to Hurricane Pam, a simulated storm that flooded New Orleans. Thirteen months later Hurricane Katrina flooded it for real. Pam had been scoped around a storm that overtopped the levees, and pre-landfall evacuation was trimmed after FEMA judged the original proposal too costly.

A crisis management exercise is a structured simulation, often a tabletop discussion, in which an organisation works through a scripted crisis scenario to test its plans, roles and decisions.

Running the hot wash, the AAR and the improvement plan

The standard sequence after an exercise is well established. The first step is a hot wash, held while the room is still together: participants and facilitators record what worked and what stalled. Evaluators add their observations, usually collected against the exercise objectives set during design.

Those notes become an after-action report (AAR). In the United States, the Homeland Security Exercise and Evaluation Program (HSEEP) formalises this as the After-Action Report/Improvement Plan, which documents "strengths, areas for improvement, capability performance, and corrective actions." Organisations outside the US typically run a local version of the same pattern, whether or not they use the acronym.

The improvement plan then turns each finding into a corrective action with an owner, a due date and a way to confirm it is done. Good practice tracks those actions to closure and uses the next exercise to check whether the fixes held. The improvement plan is where an exercise either changes something or becomes a file.

Much of the discipline here overlaps with ordinary review practice, and the same questions about what to do after an after-action review apply: who owns each action, what evidence closes it, and who notices when it quietly lapses.

A well-run exercise exposes gaps a plan hides on paper

Crisis plans read well. Exercises test whether the people named in them can actually do what the document says, in sequence, under time pressure, with incomplete information. That is valuable, and it is hard to get any other way.

An exercise puts people in the same room who may never have worked together before the real event. It exposes handoffs that nobody owns and decision rights that two agencies both assume they hold. Rehearsal turns vague responsibilities into named ones. It also builds the working relationships that leadership under pressure depends on once the event is live.

The findings are concrete and fixable, which is why they are satisfying. A logistics gap gets a new staging plan. A communications failure gets a revised call tree. Recorded properly, these become lessons learned that change the next response rather than sit in a repository.

Pull out the scenario brief from your last exercise, write down what it declared out of scope, and decide which of those exclusions the plan can least afford to leave untested. Start the Walk →

Who chose the scenario, and what did it leave out?

Every exercise begins with a scenario brief. Someone picks the hazard, sets its severity and timing, writes the injects, and decides which conditions will be treated as given. Someone also decides what is out of scope, often for reasons of budget, time or participant availability rather than risk.

The evaluation then grades performance inside that frame. HSEEP describes evaluation as assessing "the ability to meet exercise objectives," and the objectives were written for the scenario that was chosen. A weakness that the scenario never triggered produces no finding. No finding reads as no problem, even when the frame simply never looked.

This is a problem framing issue, not a performance issue. The scenario is the frame. Choose a different hazard, remove a convenient assumption from the injects, or bring an excluded phase back into scope, and a different set of weaknesses appears. The framing effect is usually discussed as a bias in how options are presented. In exercise design it operates earlier, on which problems are allowed to exist at all.

The after-action report cannot correct this on its own, because it is built from what happened in the room. An exercise cannot find a weakness its scenario never let happen.

Hurricane Pam, the storm that overtopped but never breached

The planners' original proposal covered fourteen elements, from pre-landfall evacuation through to rebuilding. According to the Senate report Hurricane Katrina: A Nation Still Unprepared (2006), FEMA officials deemed it too costly in April 2004, and Louisiana's emergency agency trimmed pre-landfall evacuation and five other issues. Witnesses said evacuation had been examined by other studies and plans, and that post-landfall response had received little attention.

The scenario was a slow-moving Category 3 storm whose surge overtopped the levees and flooded the city. The House report, A Failure of Initiative (2006), records that the exercise assumed 300,000 people would not evacuate in advance and projected more than 60,000 deaths. At the Senate hearing in January 2006, Chairman Susan Collins said the state agency had chosen "to exclude the critical issue of pre-landfall evacuation and the possibility that the levees could be breached rather than merely overtopped."

On its own terms the exercise worked. The Senate report calls it "a unique planning endeavor that resulted in functional plans" used during Katrina, including a search and rescue model. It also found that "far too many of the Hurricane Pam lessons were not applied," and that a September 2004 follow-up was postponed when FEMA could not fund $15,000 in travel. Weak follow-through was one failure. The frame was another.

When Katrina came ashore on 29 August 2005, breaches in the levee system left 80 percent of New Orleans under water at peak flooding, according to the House report. Its conclusion noted that Pam had been criticised for "its emphasis on managing the aftermath of the catastrophe and not creating initiatives that would diminish the magnitude of the catastrophe," and noted the absence of a plan to evacuate people without transportation.

Neither committee put it this way, but the 300,000 people left in the city were a premise of the exercise, not a finding of it. The workshops planned how to rescue, shelter and supply them. No review built on that scenario could have recommended reducing the number, because the frame had already fixed it.

From grading the response to questioning the scenario

The improvement plan still matters. The change is to add one check before it is signed off: reopen the scenario brief, or exercise scope statement, and read it as a list of assumptions rather than a set of settings.

Three questions do most of the work. What hazard and severity were chosen, and why that one? Which conditions were written into the injects as given rather than tested? For a critical service, what to do after a BCP test extends that question to the delivery chain, including essential partners left outside the exercise. What was declared out of scope, and was it excluded because the risk was low or because the budget was? Witnesses said Pam's evacuation exclusion rested on the belief that other studies had covered it. Pam never tested that belief.

Each excluded item then needs its own line in the improvement plan, with an owner and a status. An out-of-scope item is an untested assumption until someone tests it. The most consequential of them should set the scenario for the next exercise, and the trigger for that choice is the scenario selection meeting, before the injects are drafted.

This connects the exercise back to the crisis management plan it was meant to test. A plan validated only against a convenient scenario is validated for that scenario. Questioning the frame is the first step of the Universal Decision-Making Method, and it belongs at the end of an exercise as much as the start, because the next scenario is a decision about which weaknesses the organisation is willing to find.

---

Grant Purdy is the co-author, with Roger Estall, of Deciding (2020), and the architect of the Universal Decision-Making Method.