What we do when a prediction is wrong

Last month we argued that predictions should be open to examination. This month we show what examination looks like when the prediction is ours, and when we got it wrong.

Most forecasts are never scored

A forecast is usually judged once: on the day it is made. It is judged on how confident it sounds, who made it and how good the deck looks. By the time the outcome arrives, nobody goes back to check.

This is not dishonesty. It is the normal shape of planning. The confident calls that land get retold, and the ones that miss are forgotten. Over time, everyone believes their forecasting is better than it is.

The fix is not a better model. It is a habit: write the prediction down before anyone acts, put a date on it, and come back on that date to score it.

What reconciliation means

Reconciliation means checking each prediction against what actually happened, in four steps.

1. Record it before anyone acts. The prediction must be specific, dated and falsifiable, with the confidence stated at the time. "Sales will improve" does not count. "Weekly transactions rise within this range by this date" does.
2. Ask whether the action was taken. If the client never did the thing, the outcome tells us nothing about the prediction. That is not a miss, and we don't score it as one.
3. Score it. Each prediction is marked HIT, PARTIAL, MISS or UNRESOLVABLE. Where we stated a probability, we also score how close that probability was.
4. Name the cause of any miss. It could be a wrong baseline, a wrong mechanism, an external shock, or a correct call that nobody acted on. "Market conditions" is not an answer.

We publish our forecasts on world events and markets in a public record, the Prediction Ledger on our Insights page. Client predictions are scored the same way but kept private. We log misses as carefully as hits, because misses show where the method needs to change.

Over time, the ledger answers the question that matters most to a buyer: when Arcana says it is 80% confident, how often is it right?

Our own miss

The clearest miss we can show you this month is ours. Arcana got the call right, and we planned as if it hadn't.

The decision. In September we prepared a proposal for a downtown business-improvement board in Ontario. It was a cohort program for local businesses, written as one contract covering two quarters. We aimed for the board's October meeting and counted the revenue toward a near-term company target.

What Arcana said. We ran the decision through the protocol. It flagged that Ontario's municipal elections fall on 26 October 2026 and that the current council term ends in November. A board near the end of its mandate is unlikely to commit to a multi-quarter contract. Arcana recommended a smaller first step and warned that the October timing was at risk.

What happened. The board moved its decision to November.

How we score it.

What changed. The prediction was right. Our use of it was the problem. We now record whether each flagged risk was acted on, alongside the prediction. And we don't count revenue against a date that Arcana has marked as unlikely.

This is the miss we most want buyers to see. Most of the value of decision intelligence is lost after the analysis, when the people reading it fall back on the plan they already preferred.

The record matters more than any single call

One good prediction can be luck, and one bad one can be bad luck. Neither tells a buyer much. A record of many dated predictions, scored the same way every time, tells them how far to trust the next one.

The record also has to be built as you go. A prediction made before the outcome was known can't be created after the fact. Anyone can rebuild an analysis tool, but nobody can go back and fill in a ledger they didn't keep.

That is why we publish misses. A record with only hits is a highlight reel, not a record.

What we don't know yet

Our public record is young, and it doesn't yet meet the standard this article sets. We have used this method on client work for more than 24 months, but we only began publishing the Prediction Ledger in August 2026. As of 14 October, it reports about 13 scored forecasts, 10 of them correct, with a mean Brier score of about 0.15. These totals are provisional until every entry is reconciled.

There are two gaps. First, some forecasts were not published before their outcomes were known, including our three October Nobel calls. Second, no third party has yet confirmed that our records were made when we say they were. Until both gaps are closed, treat our totals as a working record, not as proof.

From now on, every public forecast goes on the ledger before its resolution date. Thirteen forecasts are also too few to support a calibration rate. When we have enough, we will publish one, including the decision types where we are least reliable.

Next month we will add more entries, and some of them will be misses.