Every Number Rests on a Judgment Nobody Named

A client's risk team once handed me a model with 14 input variables, three stochastic simulations, and a confidence interval that put the chance of a material loss at 2 per cent. I asked what the model's biggest assumption was. Nobody answered. Fourteen quantitative inputs, and every one of them rested on a qualitative judgment nobody in the room could name.

Guides on quantitative and qualitative decision making are everywhere, complete with scoring matrices and decision trees to help you choose the right method for your situation. Consultants and MBA programmes do well from this split; the organisations paying for the advice, less so. These are not different methods. They are different forms of the same evidence, and the cases below show what happens when nobody tests the judgment that sits beneath the numbers.

Quantitative decision making uses numerical data to inform choices. Qualitative decision making uses experience and judgment. The two are not opposites: every quantitative input rests on a qualitative assumption about what to count and how.

When quantitative vs qualitative decision making hides the real question

Long-Term Capital Management had two Nobel laureates, a team of PhD quants, and models built on 90 to 95 per cent historical correlations across two-year rolling windows. The models were correct about the past. The qualitative assumption underneath, that correlations between asset classes would stay roughly stable under stress, was never named as an assumption at all. It was treated as a feature of reality rather than a bet about the future, and the fund had leveraged that bet above 25 to 1.

When Russia defaulted in August 1998, correlations across every asset class spiked toward 1.0 simultaneously. The fund lost $4.6 billion and required a Federal Reserve-coordinated bailout. A review by the Federal Reserve Bank of Minneapolis concluded that the failure stemmed from "the separation of quantitative analysis and qualitative analysis."

That sentence is the entire argument of this article, stated by a central bank in 1999. I find it telling that the insight needed no new mathematics. It needed someone willing to say that the numbers rested on a judgment about the world that nobody had tested.

When qualitative judgment gets baked into the algorithm

When COVID-19 cancelled A-level exams in 2020, England's exam regulator Ofqual built a statistical algorithm to standardise teacher-assessed grades. Every input was a number. But someone had made a qualitative decision about what those numbers meant: that a school's historical grade distribution should constrain individual predictions. Students at historically lower-performing schools were penalised regardless of their individual ability.

The algorithm downgraded 40 per cent of teacher-assessed grades. Boris Johnson called it a "mutant algorithm." The government reversed the decision within a week, reverting to teacher-assessed grades and affecting roughly 300,000 students. A great many people blamed the maths. The maths was fine. The assumption, that where you went to school determines what you can achieve, was the judgment call nobody examined. The algorithm did exactly what it was told. The problem was the telling.

What happens when no one tests the number

On the Deepwater Horizon rig in April 2010, engineers ran a negative pressure test on the Macondo well. The drill pipe showed 1,400 psi: an unambiguous anomaly. The quantitative reading was correct. What was missing was any qualitative framework for what an anomalous result should trigger. There was no protocol and no named condition that would have forced someone to say: this number means we stop.

The crew explained the anomalous reading away with an ad hoc theory called the "bladder effect" and continued operations. The resulting blowout killed eleven people and spilled 4.9 million barrels of oil. BP's own quantitative risk assessment had estimated the most likely large spill at 4,600 barrels; the actual figure was more than a thousand times larger. The National Commission's report found that multiple cost-saving decisions were never subjected to formal risk assessment at all.

The quantitative apparatus existed. The qualitative governance, who decides when a number contradicts the plan and what happens next, did not. That is the gap risk-based decision making is supposed to close. I have seen the same pattern in quieter settings: a dashboard full of green numbers and no agreement anywhere in the organisation about which reading should make someone pick up the phone. That is a monitoring failure, and it is always a qualitative one, because the missing piece is never another sensor. It is the judgment about what the sensor means.

One test for both quantitative and qualitative evidence

In every case, the same questions would have exposed the fault before the loss. Does this evidence cover the conditions under which the assumption might fail? Does the source actually measure what we are treating it as measuring? And can the people acting on it state what an anomalous result would look like?

These are not quantitative questions or qualitative questions. They are evidence questions, and in the Universal Decision-Making Method they apply to a regression output and a site visit in exactly the same way. I do not sort evidence into hard and soft bins, and I have never met a decision that failed because someone used the wrong category. The failure is always upstream: in the assumption that was never tested.

Roger Estall and I put it plainly in Deciding: the significance of any piece of evidence depends on its influence and your confidence in it, and those two parameters "cannot be combined in a mathematically valid way." Even the framework's own assessment is qualitative, and that is not a limitation.

That 14-variable model I mentioned at the start? I asked the team one question: under what conditions would these inputs change? They identified six variables that would move in a downturn. None had been stress-tested against a downturn scenario. Nobody had even thought to ask. The team had spent months debating quantitative vs qualitative approaches. They were having the wrong argument. The fault line is the assumption nobody named. That is what data-driven decision making actually requires.

---

Grant Purdy is the co-author, with Roger Estall, of Deciding (2020), and the architect of the Universal Decision-Making Method.