How PunditScore works
Transparency over black-box scoring. Evidence over reputation. Comparability over anecdote.
Every PunditScore traces back to a dated claim, a real-world outcome, and the reasoning in between. This page describes exactly how the number is produced — and what it does and doesn’t claim.
01Four stages, kept separate
A pundit is assessed as three separable questions — did they say it, did it happen, how good was the call — so each can be checked on its own. The cardinal rule across all of them: retrieve first, reason second. The model never recalls a claim or an outcome from memory; it only works from retrieved sources.
Search the open web for who the person is and the predictions they've publicly made — pulling the actual source documents, never the model's memory.
Read only those retrieved sources and pull out dated, attributable, falsifiable claims — each with a verbatim quote and its source.
For each claim, search for what actually happened and classify the real-world outcome against that evidence.
The model judges each prediction against its resolved outcome; the KPIs and the composite are then computed by a fixed formula — not emitted by the model.
02The four KPIs
Every score decomposes into four measures. Directional, Timing, and Revision are the average of the per-prediction judgments across an analyst’s resolved predictions; Market Impact is judged at the analyst level.
03The composite is a formula, not a guess
The headline PunditScore is computed from the four KPIs by a fixed, versioned weighting — it is not a number the model emits. That means anyone can re-derive it by hand, and the same inputs always give the same score.
Weights are normalized by analyst type. For Macro & structural analysts, whose calls run over long horizons, the Timing weight is lowered to 0.10 (from 0.20) and shifted to Directional — a decade-out structural call isn’t judged on timing like a near-term operational one.
04Score bands
05How each prediction resolves
06Where we say “we don’t know”
A score is only as strong as the resolved evidence behind it. When fewer than 3 of an analyst’s predictions have actually resolved, the scorecard is marked Provisional rather than presented as firm — many pundits (especially futurists) make calls that simply haven’t come due yet, and we say so instead of inventing confidence. Likewise, a prediction whose horizon has already passed can never sit as “pending”: it must resolve or be flagged unresolvable.
Scoring is pinned for reproducibility — a fixed model version, zero-temperature sampling, and a versioned weighting config — so the same analyst returns the same score for the same reasons. The archetypes we normalize by: Operational & industry · Macro & structural · Financial & Wall Street · Institutional & advocacy · Investor-banker & operator-prognosticator.
07What a PunditScore is not
It is not an absolute verdict on a person. It reflects the public, checkable record we could find — paywalled research, private notes, and off-the-record calls are not scored, so analysts whose work is mostly private will look thinner than their reputation. Scores are relative and evidence-bounded, calibrated against each other rather than against absolute truth. A score you can’t interrogate would be a failure of the design — so every number here opens to its underlying quote and outcome.