The Data Was Right Every Time. The Decision Was Wrong for the Same Reason.

In January 2012, JPMorgan's Chief Investment Office reported its portfolio risk within approved limits, backed by a Value-at-Risk model that had been recalibrated the same month. Three months later the same portfolio had lost $6.2 billion.

The data was not the problem. That is the pattern worth studying in data driven decision making examples: not the cases where data was absent, but the ones where it was right and the call was still wrong.

I have seen that sequence play out dozens of times across industries: the right data and the wrong call. Most collections of data driven decision making examples show organisations that lacked data and improved once they gathered more. Those are reassuring and almost entirely useless to anyone who already has the dashboards and still cannot decide. For anyone who has sat in a room with a full dashboard and a committee that still cannot commit, that logic describes a different planet.

The examples worth studying are the ones where the evidence was sound and the decision was still wrong, because the failure sits in assumptions nobody named and evidence tests nobody ran.

Data-driven decision making is the practice of basing decisions on evidence rather than intuition alone. The practice is only as sound as the assumptions the evidence is supposed to test.

JPMorgan Changed the Model to Fit the Answer

The Chief Investment Office built a large synthetic credit derivatives portfolio. As losses mounted in early 2012, management did not investigate whether the positions were genuinely hedging risk. Instead, they approved a change to the Value-at-Risk model that cut reported risk roughly in half. Under the original model, the portfolio would have triggered more than 330 risk-limit breaches; under the replacement, it showed none. CEO Jamie Dimon dismissed the early signs as "a tempest in a teapot." The Senate Permanent Subcommittee on Investigations later documented the full sequence of internal communications showing management knew of the model change and its effect on reported risk.

The assumption underneath was straightforward: these positions reduce overall firm risk. The evidence test was available in real time, because the portfolio was losing money while the positions it was supposed to hedge kept moving against it. If the positions were a genuine hedge, losses in one leg would have been offset by gains in the other, but they were not. Instead of running that test, the team adjusted the measurement instrument until the readings looked comfortable. The traders whose positions survived scrutiny and the risk managers whose limits stayed clean both had reason to prefer the new model.

The fix was available: name the assumption explicitly and design a parallel test. If the VaR model is changed, the original model runs alongside it for ninety days and any divergence above a stated percentage goes to the board. A monitoring trigger, written down before approval, absent because nobody had surfaced the assumption the model depended on.

Click to expand

Wells Fargo's Data Driven Metric Measured the Wrong Signal

Wells Fargo's retail banking division tracked cross-selling metrics with near-religious devotion. The internal target, known inside the company as "eight is great," aimed for eight products per household. The data showed steady growth: accounts opened and revenue per customer kept climbing. Between 2002 and 2016, approximately 3.5 million accounts were opened without customer authorisation. The US Department of Justice settlement cost $3 billion, over 5,300 employees were fired for sales practice violations, and the CEO resigned.

The data was accurate and accounts were in fact being opened, but the assumption underneath the metric was not tested: that growth in accounts-per-household meant customers valued additional products. The metric measured the output and never questioned the input. Branch managers hitting cross-sell targets had no incentive to ask what dormant accounts meant. Internal complaints about fraudulent account-opening practices had surfaced years before the public scandal broke. Of the three data driven decision making examples here, this one had the most warning time and the least excuse.

Every signal that would have caught the failure was already available inside the bank: account dormancy within ninety days of opening, and customer complaint rates that kept climbing. Nobody designed the monitoring to watch those signals because nobody had named the assumption the headline metric rested on. A data-driven culture that cannot question its own lead metric is a reporting culture with better formatting.

Name the assumption your lead metric rests on, before the next committee treats the number as the verdict. Start the Walk →

Norfolk Southern Had the Data and the Wrong Threshold

On 3 February 2023, a Norfolk Southern freight train carrying hazardous materials derailed in East Palestine, Ohio. Thirty-eight cars left the track, and five tank cars carrying vinyl chloride were later vented and burned in a controversial decision that displaced residents across the surrounding area. Wayside hot-box detectors along the route had been measuring a rising bearing temperature on the car that derailed. The last detector before the failure site recorded the bearing at 103 degrees Fahrenheit above ambient, but the alarm threshold was set higher and it fired too late, leaving the crew less than a minute between the alert and the derailment. The NTSB investigation report documented that other railroads used lower thresholds for the same class of detector.

Bearing temperature was measured accurately every time a train passed the detectors, and the system was functioning as designed. The decision embedded in the monitoring design was wrong: the threshold at which a temperature reading became actionable had been set on assumptions about what constituted danger, and those assumptions were never tested against peer benchmarks or actual failure data. Operations managers who measured on-time performance had every reason to leave a threshold that kept trains moving.

This case is as clean an example as I have seen of a monitoring-design failure. The evidence was present and accurate, but what was missing was a threshold calibrated against actual bearing failures and a review mechanism that compared the setting to what peer railroads used. Monitoring that never revisits its own assumptions decays into a formality, which is precisely what happened here.

What Data Driven Decision Making Examples Actually Teach

Every organisation in these three cases had data and monitoring, yet each failed because an assumption between the data and the decision went unexamined.

JPMorgan assumed the portfolio hedged risk; Wells Fargo assumed account growth meant genuine demand. At Norfolk Southern the assumption sat inside the alarm threshold itself, calibrated against failure patterns that nobody had revisited since commissioning. In each case the dashboards, the models, and the thresholds were all designed to watch the metric, not to question what the metric stood on.

In every case the assumption was testable and nobody tested it, and preventing that failure does not require a specialist. Someone in the room needs to ask, before the decision is approved, which assumption is carrying the weight, and what evidence would collapse it, and then write the answer down alongside the signal that would prove it wrong.

Roger Estall and I wrote Deciding because we had both watched competent organisations collect data with enormous discipline and still make the wrong call. The failure was never the collection; the pattern across these data driven decision making examples is the same one we documented: data that tests assumptions is useful, and data that replaces judgement with a model output is dangerous. The wider case for data-driven decision making starts there, with the assumption that links the evidence to the call, and the question of whether anyone has tested it.

---

Grant Purdy is the co-author, with Roger Estall, of Deciding (2020), and the architect of the Universal Decision-Making Method.