Your risk register is not a model

Somewhere in your organisation there is a spreadsheet with a row that reads something like: Ransomware affecting core ERP. Likelihood 3. Impact 4. Score 12. Amber. Owner: IT Director. Mitigation: ongoing.

That row has survived three audits, two reorganisations and one actual incident. It has never caused a decision to be made differently. And it is presented to the board as risk management.

Douglas Hubbard's The Failure of Risk Management and, with Richard Seiersen, How to Measure Anything in Cybersecurity Risk make a case that deserves more attention than it gets in security teams. The dominant method in our profession, ordinal scoring on a colour-coded matrix, is not a simplified version of quantitative risk analysis. It is a different thing, and there is little evidence it works.

Three specific problems, not a vague complaint

The critique is often reduced to "heat maps are bad". It is more precise than that.

The arithmetic is invalid. A likelihood of 3 is not three times a likelihood of 1. Ordinal scales are ordered labels, and multiplying labels produces a number with no defined meaning. Tony Cox's analysis in Risk Analysis (2008) went further and showed that under plausible conditions a risk matrix can rank risks no better than chance, and can assign a lower rating to a quantitatively larger risk. The mechanism is range compression: a risk with a plausible loss of $300,000 and one with a plausible loss of $30 million both land in the "4" band, and the distinction that matters most to the business is destroyed at the point of data entry.

The language is ambiguous. Sherman Kent documented this inside the CIA in the 1960s. Give a group of analysts the word "likely" and their implied probabilities spread across most of the number line. Nothing has changed. When five assessors score the same threat as "possible", they are not agreeing; they are using a word that hides their disagreement. A calibrated estimate of 8% can be wrong. "Possible" cannot be wrong, which is precisely the problem.

The output does not aggregate. You cannot sum ordinal scores. So the register can never answer the only two questions an executive committee actually asks: what is our expected annual loss from cyber events, and does this $2 million of proposed spend reduce it by more than $2 million? A method that cannot answer those questions is not a decision-support tool, whatever else it may be.

Why it survives

Hubbard's most uncomfortable observation is not about mathematics. He calls it the analysis placebo: structured, effortful, professional-looking processes reliably increase participants' confidence in their conclusions without increasing the accuracy of those conclusions. A workshop feels productive. Consensus feels like convergence on truth. Colour feels like clarity.

Because almost no organisation ever measures whether its risk management improved outcomes, the placebo is never detected. Hubbard pairs this with what he calls the risk paradox: the more significant the decision, the more likely it is to be assessed by the softest method available.

The register is not the villain

Here is the part that surprises people who have only heard the summary. Hubbard is not anti-spreadsheet. How to Measure Anything in Cybersecurity Risk teaches its core method in Excel, with downloadable workbooks. The Monte Carlo simulation at the heart of it is a few hundred rows of formulas that any analyst can build before lunch.

The problem was never the tool. It is what we put in the cells, and the fact that nobody comes back. A register that is scored once a year in ordinal units, never back-tested against events, and never used to price a control is a compliance artefact: a museum, not a model.

The good news is that the fix is a substitution, not a rebuild.

One row, converted

Take the ransomware row above. All figures below are NZD. Replace "Likelihood 3, Impact 4" with two calibrated statements from the people who know the environment:

• Probability of at least one such event in the next 12 months: 8%. Anchored on published base rates (the Cyentia Institute's Information Risk Insights studies and the Verizon DBIR give event frequencies by organisation size and sector), then adjusted for what you know about your own controls.
• If it happens, the loss falls between $250,000 and $12,000,000, with 90% confidence. Wide, because your uncertainty is wide. That is the point: the interval is honest, whereas "4" is not.

Model the loss as lognormal, which fits the skew of real incident data far better than a symmetric range. In Excel:

Fill D2 down 10,000 rows. Each row is one simulated year: mostly zero, occasionally expensive. Then:

• Expected annual loss: =AVERAGE(D2:D10001), which returns about $277,000
• Exceedance probability at any threshold placed in F2: =COUNTIF($D$2:$D$10001,">"&F2)/10000

Plot those exceedance probabilities against the thresholds and you have a loss exceedance curve:

Notice what the ordinal version concealed. The median loss given an event is $1.7 million; the mean is $3.5 million. The distribution is dominated by its tail, and the entire tail was living inside the digit "4".

Now the register can be used

Draw one more line on the same chart: the board's stated tolerance. Suppose they will accept no more than a 5% annual chance of losing more than $1 million. Our single risk sits at 5.4%, which on its own is marginally over the line. That is a conversation with a conclusion, which is more than the amber cell ever produced.

Then price the control. A proposed programme costs $400,000 a year and, on the team's calibrated judgement, cuts the annual probability from 8% to 3% and pulls the upper bound of the loss range from $12 million to $7 million. Rerun the same two formulas: expected annual loss falls from $277,000 to $66,000. The reduction is roughly $211,000 a year against $400,000 of spend.

The maths says no. Or rather, it says that break-even is around $210,000, so either negotiate the price down, find a control that also compresses the tail, or spend the money on the risk next in the queue. Hubbard calls this return on control. It is an ordinary capital allocation argument, in the currency the board already uses, and no register built on 1-to-5 scoring can produce it.

The objections, taken seriously

"We don't have enough data." You have less than an actuary would like and more than you think. Base rates are published. Internal ticket, incident and downtime data exist. And note the asymmetry: the ordinal score was produced from the same evidence base, then processed with worse mathematics. Sparse data argues for Bayesian updating, not for abandoning quantities. Hubbard's rule of five is a useful demonstration of how little is required. With a random sample of just five, there is a 93.75% chance the population median lies between the smallest and largest values observed.

"This is false precision." Precision and accuracy are different. A 90% interval spanning two orders of magnitude is an explicit statement of ignorance. "Impact: high" is an implicit one, and cannot be audited, aggregated or proved wrong.

"Our estimates are just guesses." They are, until they are calibrated. Calibration training, meaning scoring people against known answers until their 90% intervals actually contain the answer 90% of the time, is measurable, teachable and, in Hubbard's reported results, effective for the large majority of participants within a day. Almost everyone starts overconfident. Nobody's ordinal scoring has ever been calibrated at all.

"The auditor wants a matrix." Then render one from the model. Deriving a heat map from a quantitative result is trivial. The error is using the heat map as the model.

Where to start on Monday

1. Take the top ten rows of the existing register. Leave the other three hundred alone.
2. Run a half-day calibration session with the people who will supply the estimates.
3. Replace each row's likelihood and impact with a probability and a 90% loss interval. One-for-one substitution: same rows, same owners, better units.
4. Build the simulation described above and produce a single loss exceedance curve for the ten.
5. Ask the executive committee for one number: the loss and likelihood at which they would rather spend than accept. Draw it on the chart.
6. Record every estimate with a date, and check the frequencies against reality next year. This is the step everyone skips, and the only one that turns a register into something that learns.

None of this requires a platform, a consultant or a new framework. It requires abandoning the belief that a colour is an answer.

If you would rather not do the first pass alone, converting those top ten rows is what a quantitative risk analysis engagement covers, and control effectiveness testing is how you check the evidence the estimates are resting on.

Sources

• Douglas W. Hubbard and Richard Seiersen, How to Measure Anything in Cybersecurity Risk, 2nd edition, Wiley, 2023
• Douglas W. Hubbard, The Failure of Risk Management: Why It's Broken and How to Fix It, 2nd edition, Wiley, 2020
• Louis Anthony (Tony) Cox Jr., "What's Wrong with Risk Matrices?", Risk Analysis, 28(2), 2008
• Sherman Kent, "Words of Estimative Probability", CIA Studies in Intelligence, 1964
• Cyentia Institute, Information Risk Insights Study; Verizon, Data Breach Investigations Report