Data-Driven Decision Making Still Needs a Method

Every organisation I work with says it practises data-driven decision making. Most of them mean they have dashboards. A dashboard is a picture of what happened yesterday; it does not tell you what to do about tomorrow. The moment someone has to commit resources, sign a contract, or shut a plant, the dashboard sits in the room like a well-dressed bystander.

Most teams have more than enough data. What they lack is any discipline for deciding what the data actually tests. The verdict belongs to nobody in particular. The dashboard produced it. The model endorsed it. The committee reviewed it. Nobody decided.

Data-driven decision making is the practice of using evidence to test the assumptions beneath a decision, then acting once you know enough to proceed, and designing monitoring that catches the moment those assumptions stop holding.

That definition sounds straightforward. It is. The difficulty is that most organisations have built elaborate machinery to avoid doing exactly this.

Click to expand

What the Numbers Are For

Numbers are for testing whether the assumption underneath a recommendation can still hold. They are not for creating the mood of rigour in a meeting room.

I worked with an organisation that built its strategic plan around a set of explicit assumptions, each rated for volatility. Revenue forecasts, supplier stability, regulatory expectations: every one was written down, and every one carried a note saying how fast it could change. When an assumption was rated high-volatility, it triggered a monitoring rule. If the market moved past a named threshold, the assumption reopened and the plan was reviewed.

That is what a data-driven approach looks like when the phrase means something. The data did not predict the future. It forced the planning team to say what had to remain true for the plan to survive, and it gave them a signal to watch instead of a forecast to worship.

Compare that with the usual approach: a spreadsheet full of projections, each carrying decimal precision that flatters the forecaster and anaesthetises the reader. The decimals signal confidence. They do not signal that anyone has asked what happens when the input moves. A plan built on unrated assumptions is not a plan. It is a bet that nobody has agreed to take.

Most organisations generate far too many columns of information, most of which nobody uses. The volume creates a feeling of coverage without the discipline of asking what the numbers actually test. If a forecast does not name the assumption it rests on and the condition that would invalidate it, the forecast is decoration.

Organisations that try to fix this by launching a data-driven culture programme usually add more dashboards without adding the two tests that matter: who may challenge the number, and what assumption does it stand on.

The harder question is whether data-driven leadership means pointing at a dashboard or owning the assumption behind it. Most leaders do the former and call it evidence-based.

Surface the Assumptions Behind the Data

A number that looks factual today can become an assumption overnight. Production cost and freight can be known this morning. The selling price still rests on what happens before the first shipment lands. If resin, freight, or exchange rates can move, the number is only as sound as the conditions beneath it.

The useful discipline is to ask five questions about the evidence: how complete it is, how accurate, how reliable, how intelligible, and how quickly it could change. Those questions sort evidence from decoration. A quarterly report can score high on completeness and intelligibility while scoring badly on reliability if nobody has checked the source data against reality for three years.

Facts decay. A supplier price locked in last year is a fact. A supplier price locked in three years ago, when the contract assumed a logistics environment that no longer exists, is an assumption wearing the clothes of a fact. The Universal Decision-Making Method puts this question at the centre: what are we assuming, and how significant is each assumption to this decision? That reframes data from a trophy to a tool. It also shortens meetings, because about half the information people bring to a decision room is there for atmosphere rather than evidence.

An evidence audit is not a pleasant meeting. People prefer to treat information as settled, because settled feels professional. But the professional move is to mark which pieces of information are still genuine knowledge and which have quietly turned into hope. Assumptions in decision making are mistaken for facts more often than most teams want to admit.

The debate over quantitative vs qualitative decision making runs into the same wall. Every quantitative input rests on a qualitative judgment about what to measure and how. Arguing about the method is easier than testing the assumption the number rests on.

When Models, Dashboards, and Forecasts Pretend to Decide

A model is an assistant with no judgment. It can show what follows if certain conditions hold. It cannot tell you whether those conditions still hold right now, in your building, with your contracts.

I worked with an organisation where the auditors had given a clean report for years running. The compliance dashboard showed green. Everybody relaxed. The problem was that the instruments behind the dashboard had drifted out of calibration, and the readings kept arriving neatly because drift is not visible when nobody questions the readings themselves. A clean report was treated as truth when it was only a reflection of yesterday's calibration schedule.

That is how dashboards fail. They do not lie. They report what the instrument captures, and nobody asks whether the instrument still captures the right thing.

The Uber disruption makes the same point from the other direction. Every signal was visible before ride-sharing arrived: customer dissatisfaction, pricing complaints, and the spread of smartphone mapping. The taxi industry did not lack data. It lacked anyone willing to synthesise what the signals meant together. More data does not help if nobody joins the dots into a question that forces a choice.

The failure is not ignorance. It is interpretation. Having all the data in the room and failing to make the call is worse than having half the data and owning the judgment. Organisations that collect compulsively without deciding promptly are not being rigorous. They are being evasive.

Deciders must be wary of treating calculations or mathematical models intended to assist with making decisions as if the results are the decision. A printout is marvellous cover for anyone hoping nobody asks a second question. When the room treats the model as the Decider, data paralysis follows: the team collects endlessly because nobody is willing to own the call the data is supposed to inform.

When Data Conflicts or Goes Missing

Most advice on this subject assumes you have clean data and enough of it. In practice, the data often conflicts, arrives late, or does not exist for the variable that matters most.

That is not a failure. It is the normal condition. The method handles it with three levers. First, you can seek more information, but only if the cost and delay are proportionate to the decision. Second, you can modify the decision to reduce its exposure to the uncertain variable. Third, you can choose a different option altogether, one that rests on assumptions you can actually test.

The sports-match example makes this plain. A risky play attempted while the scores are level can lose the match. The same play attempted once the lead is comfortable costs almost nothing if it fails. Sometimes the right response to missing data is not to find the data but to change the stakes so the gap matters less. Timing is not a concession to uncertainty. It is one of the tools for reducing significance.

People dislike this because they want a universal evidence threshold, preferably printed in a handbook and blessed by a committee. The world is under no duty to provide one. How much is enough information to make a decision depends on what the decision serves and what happens if the call is wrong.

Data-Driven Decision Making Does Not End at Approval

The assumptions that carried the call are still live. Any one of them can begin to decay the moment the ink dries.

Monitoring is not a compliance afterthought. It is part of the decision itself. I want the review rule written before money moves or product ships: a named signal, a named person, and a threshold that triggers review. If the signal moves past that threshold, the person acts. If nobody owns the trigger, monitoring turns into a comfort object, and organisations are very good at filing comfort objects carefully.

A dashboard that shows you last month's figures without naming the assumption each figure was supposed to protect is noise. I once asked a board what would have to change in their operating environment to invalidate their current strategy. Nobody could answer. They had forty metrics and no review trigger.

The method specifies monitoring before commitment because that is when assumptions are most visible. After approval, the room scatters, memory edits the reasoning, and the original conditions start to look like settled truth rather than the live hypotheses they actually were. Monitoring a decision means tracking the assumption that could still injure it, not re-running the dashboard that endorsed it.

We manage risk to create value through enhancing the decisions we make. We do not manage risk to create reports.

Data-Driven vs Intuition: A False Binary

The debate between data and intuition assumes they are rival systems. They are not. Intuition is pattern recognition compressed by experience. It is fast, and it is often right in familiar territory. It fails on novel decisions, which is exactly where the stakes tend to be highest.

The useful question is not whether to trust data or intuition. It is whether the assumptions your intuition is making have been stated plainly enough for someone else to challenge them. Data tests assumptions. Intuition generates them. Pretending one can replace the other produces either analysis paralysis or overconfidence, depending on which camp wins the room.

An operations leader who says "I just know this supplier will fail" is surfacing compressed experience. That experience may be sound or it may be out of date. The value of making the assumption explicit is not that it proves the intuition wrong. It gives the room something testable instead of something they either accept on faith or reject on principle. The longer version of that argument, with the J.C. Penney and firefighter cases, is in intuitive decision making.

A Framework for Data-Driven Decision Making

Most frameworks for data-driven decision making are collection plans: gather metrics, build dashboards, assign KPIs. Even the balanced scorecard, which was designed to go beyond financial numbers, usually degenerates into exactly this. They treat the data pipeline as if it were the decision process. It is not.

A framework that serves the Decider starts by framing the decision, then surfaces the assumptions the options rest on, tests those assumptions against available evidence, and specifies monitoring before commitment. The Universal Decision-Making Method does exactly that. It does not prescribe what data to collect. It asks what the decision needs to survive, and it lets the evidence gaps show honestly instead of burying them under metric volume.

Collection-first frameworks also create a perverse incentive. The more metrics an organisation tracks, the more rigorous it appears, and the less likely anyone is to ask whether the metrics test anything that matters. The framework becomes a performance of data use rather than an instrument of it.

I wrote a longer piece on how a data driven decision making framework should start at the decision rather than the data pipeline. The argument there is that thresholds for action matter more than volume of evidence.

Decision Quality Is Not Outcome Quality

Good decisions can still produce bad outcomes, and bad calls sometimes get rewarded. Judging the decision by the outcome is the most common evaluation error in organisations that treat data as a scorecard.

If the assumptions were reasonable, the evidence was tested, and the monitoring was designed before commitment, the decision was sound. If the outcome disappoints, the monitoring should trigger a review of what changed. That is not a failure of the process. It is the process working as designed.

Organisations that punish good decisions with bad outcomes train their people to avoid commitment. The safest move becomes asking for more data, delaying the call, and hoping the decision outlives the decision-maker's tenure. That is career-driven, not data-driven. Decision quality lives in the discipline applied before the outcome arrives.

The evaluation question is not "did it work?" It is "did we know what it depended on, and were we watching?" If the answer is yes and the outcome still disappoints, the process did its job. The monitoring tells you what shifted. That information becomes the starting point for the next decision, not a performance review of the last one.

---

Grant Purdy is the co-author, with Roger Estall, of Deciding (2020), and the architect of the Universal Decision-Making Method.