When Decision Analysis Becomes Camouflage

On 8 March 2023 Silicon Valley Bank sold $21 billion of securities at an after-tax loss of $1.8 billion and announced a $2.25 billion capital raise. By the next day depositors were trying to pull more than $40 billion, about 85 per cent of the bank’s deposit base. SVB had stress tests and interest-rate models. It did not lack decision analysis under uncertainty. It had analysis that management bent until it stopped telling the truth.

Decision analysis under uncertainty is the practice of testing assumptions to reach sufficient certainty, not a way to hide the decision behind arithmetic. The moment people start treating the output as if it has made the choice for them, the analysis becomes camouflage. In my experience, that is when management starts protecting its story and committees start avoiding ownership. Analysts then answer the wrong question because it is easier to model than to decide.

When the model stops telling the truth

The Federal Reserve’s review of Silicon Valley Bank is blunt. SVB removed interest-rate hedges, moved to less conservative stress-testing assumptions, and behaved as if a concentrated deposit base would stay put. Those were the assumptions carrying the bank’s position, not technical footnotes.

That was a decision failure expressed through a model, and it is the broader problem in decision making under uncertainty. A model can estimate exposure to rising rates. It cannot tell you whether frightened depositors will behave nicely so management can keep its narrative intact. Once the assumptions were adjusted to preserve a preferred answer, the analysis stopped testing reality and started defending management from reality. The usual plea at that point is for more data. What is usually missing is the nerve to name the assumptions already smuggled into the model.

Why bad assumptions survive good models

Click to expand

Ofqual walked into the same trap in 2020. With exams cancelled, it built a standardisation model to moderate teacher-assessed grades across 738,418 A-level entries. The model assumed each centre’s attainment would show continuity with prior years and that prior attainment would remain a stable guide to results. The outputs looked orderly. The country saw something else: students receiving grades that were statistically neat and individually absurd.

The Ofqual interim report later conceded that the result did not command public confidence. I am not surprised. The model answered a narrow technical question about distributional consistency. It did not answer the decision that mattered, which was how to preserve trust in grades during an abnormal year. Committees do this all the time. They pick the question a model can answer, then delegate the real decision to the output and call that rigour.

If the Purpose is wrong, the model is just a faster way of getting the wrong answer. Roger Estall and I built the Universal Decision-Making Method because of that recurring fraud. I have seen land managers treat a wildfire model’s output as if the fire had endorsed it. The calculation can estimate spread if its assumptions hold. It cannot tell you that those assumptions deserve your trust today.

The analysis still needs a Decider

NASA’s handling of Starliner in August 2024 shows the opposite pattern. After helium leaks and thruster trouble on the flight to the International Space Station, NASA and Boeing ran tests and pulled in outside propulsion experts. Then NASA decided to bring Starliner back without crew. That mattered because the agency had already done the analysis and still refused to pretend the analysis had settled the question. It did not let programme pride or sunk cost masquerade as certainty.

NASA’s 24 August 2024 announcement said the teams had completed extensive analysis, but the remaining uncertainty and lack of expert concurrence did not meet human-spaceflight safety requirements. That sentence is better than a shelf of decision textbooks. Sufficient certainty existed for an uncrewed return. Same telemetry, different consequence, so the crew did not ride home on it. That is also why I keep monitoring attached to the decision itself. Where the context keeps moving, the issue starts to resemble decision making under deep uncertainty, because assumptions can fail while the operation is still underway.

How I use this in a boardroom

Boards like starting with the model because the model cannot argue back. I start by asking what decision we are actually taking, and for what Purpose. If nobody can answer that plainly, the analysis is already wandering. Once the decision is named, I ask which assumptions inside the model are carrying the weight. Usually there are only a few that matter. The rest is scenery.

At SVB the load-bearing assumptions were deposit stickiness and rate exposure. At Ofqual they were historical continuity and the legitimacy of treating a student as a member of a cohort. Once the load-bearing assumption is named, half the apparatus is exposed as scenery. That is where decision analysis under uncertainty becomes useful again, because the room stops admiring the machinery and starts examining what the machinery is being used to disguise.

If the one assumption carrying the decision still looks weak, I either test that assumption or change the decision. I do not commission another omnibus report. The room resists this because testing one assumption is less impressive than modelling twelve scenarios, and less billable too. But one tested assumption outranks a shelf of unowned analysis every time. That is what Roger Estall and I set out in Deciding. If you want the plain-language version, read making decisions with uncertainty.

---

Grant Purdy is the co-author, with Roger Estall, of Deciding (2020), and the architect of the Universal Decision-Making Method.