How PunditScore works

Transparency over black-box scoring. Evidence over reputation. Comparability over anecdote.

Every PunditScore traces back to a dated claim, a real-world outcome, and the reasoning in between.

In brief technical terms, PunditScore uses large language models to extract each dated claim, judge it against retrieved evidence one claim at a time, and grade it on a fixed rubric. Checks written in code catch what code can catch, and a published formula turns the grades into a score, so every step can be inspected and audited.

This page describes exactly how the PunditScore is produced — and what it does and doesn’t claim.

01Seven stages, kept separate

A pundit is assessed as three separable questions — did they say it, did it happen, how good was the call — so each can be checked on its own. The rule across all of them: retrieve first, reason second. Every model call is handed retrieved documents and told to work from them, not from what it remembers.

Each judgment gets its own model call. An earlier version asked one call to rule on a whole pundit’s claims at once, and it anchored on its first answer and repeated it: one analyst had 44 conflict flags sharing two wordings. Every judging stage now handles one claim per call.

Every model stage uses the same model, OpenAI’s gpt-4o-mini, reached through a name that OpenAI can update rather than a dated snapshot. One model family therefore both extracts a claim and judges it. We say more about what that risks in §10.

02Checks run in code, not by the model

A model can be told a rule and still break it. Where a rule can be written as code, it is, and the model’s output is checked or overwritten rather than trusted. These run on every pundit:

Two judgments stay with the model because no code can make them: whether a sentence is a forecast at all, and what the evidence says happened. If the per-claim forecast filter errors, the claim is kept rather than lost, so a failure there adds claims rather than removing them.

One check is missing, and it matters. The quote each claim is extracted with is not yet checked against the document it came from. Tools exist for that job: small models now check whether a document supports a sentence about as accurately as GPT-4, at a fraction of the cost^[1]. They answer does this text say that, which fits a quote. They do not answer did this happen, so they cannot replace the outcome judge.

03What gets measured

Three measures feed the score, and each is judged against what actually happened. Directional, Timing, and Revision are the average of the per-prediction judgments across a pundit’s resolved predictions.

Alongside those we publish context — pundit-level characteristics that are deliberately not part of the score. How widely cited someone is, or what commercial stake they hold, tells you how to read a call; neither is evidence that the call was right. Folding them into the number would make one score mean two different things.

04The composite is a formula, not a guess

The headline PunditScore is computed from the three accuracy measures by a fixed, versioned weighting — it is not a number the model emits. That means anyone can re-derive it by hand, and the same inputs always give the same score.

Weights are normalized by analyst type. For Macro & structural analysts, whose calls run over long horizons, the Timing weight is lowered to 0.15 (from 0.25) and shifted to Directional — a decade-out structural call isn’t judged on timing like a near-term operational one.

One rule inside that formula is worth stating rather than leaving you to find. Timing is scored on every resolved prediction, including the ones whose direction was wrong. So a wrong call usually loses points twice: 77% of wrong-direction calls also score zero on Timing. Directional therefore pulls harder on the composite than its 0.50 weight suggests. We would rather publish that than quietly carry it.

05Score bands

These cuts are not fixed. They are placed at set distances from the mean of the 49 tracked pundits who have at least 10 resolved predictions, measured in that group’s own spread (mean 57.1, standard deviation 7.6). “Cohort” here means that whole tracked panel of 49, not other analysts of the same type. So a band says where a pundit sits among the people we track, not against a universal standard, and the cuts move as the panel grows. Cut points and band edges are rounded to one decimal place; the comparison itself runs on the unrounded values. That rounding is also why this spread reads 7.6 here and 7.6 in §06: the bands are cut from published composites, which are whole numbers, while §06 measures the underlying per-call scores before any rounding.

The middle band is the widest of the five, but widest is not the same as most: on today’s panel about 69% of banded pundits get a label that says above or below, and only 15 of 49 sit at the average. We publish that share because the labels are only worth as much as §06 says they are: at 74% agreement a five-way split is supportable, where at the 41% we were reporting before this corpus was collected it was not. If that number falls again, the bands should get coarser rather than the caveat longer. A pundit with fewer than 10 resolved predictions gets no band at all: too few resolved calls cannot tell a poor forecaster from an unlucky one.

06How much of the number is real

This is the section most scorecards leave out. A composite is an average, so the spread we show between pundits is part skill and part luck, and those two can be measured apart. Across the 49 pundits with enough resolved calls to be scored:

Read that plainly: if one pundit shows 6 points above another, expect about 69% of that gap — near 4.2 points — to be a real difference in judgment, and the rest to be the luck of which calls they happened to make. That 69% is a share of variance, which is the figure that tells you how far to discount a gap. Compared as spreads rather than gaps the same data reads 83%, because 6.4 is 83% of 7.6. Both numbers are true and they answer different questions, so we say which one we mean.

Two more figures, and they are not the same figure. Sampling noise across the panel has a spread of 4.2 points — that is the term which, with skill, adds up to the 7.6 we display. A single pundit’s own score carries an uncertainty of about 3.6 points at the median record length of 80 resolved calls, more for a thinner record and less for a fuller one. Either way, two people a few points apart are not distinguishable. Split a pundit’s own record in half and the halves agree on the ranking at 0.59 out of 1; stepped up to full record length that is roughly 0.74, which is the number comparable with the 69% above. Two independent routes to the same answer.

One caveat on the 6.4, and it cuts against us. That figure is what persists across a pundit’s record once luck is subtracted, which is not quite the same as skill. A steady habit of making easy, safe calls persists exactly as well as good judgment does, so it survives the subtraction and gets counted as judgment. The real skill spread is therefore somewhere below 6.4, and the 69% is a ceiling rather than a floor. Separating the two needs an outside difficulty measure for each claim — a market price or a historical base rate — which is the work described in §11, not more evidence of the kind we already have.

The standard reference on scoring forecasts says it directly: scores compare cleanly only when they cover exactly the same forecasting situations, and otherwise how predictable the questions were gets tangled with how good the forecaster was^[2]. Forecasting benchmarks hit the same wall when forecasters answer different questions. Their fix is to estimate how hard each question was from everyone who answered it, and then credit each forecaster for beating that difficulty^[3]. The method needs questions shared across forecasters, directly or through a chain of them. On our panel no two pundits have yet been graded on the same claim, so there is nothing for it to work with. We tried: a difficulty model fitted to our data simply hands back each pundit’s raw average.

The sampling half of this is a sample-size problem and more resolved calls fix it: the median scored pundit has 80 resolved calls, and it takes about 103 before four fifths of their score is signal. That is the same order of record a forecasting tournament needed: there, past accuracy became the best guide to future accuracy once about 20 questions had resolved, and kept adding information through 80^[4]. Everyone in that tournament answered from one shared pool of questions, so it could adjust for difficulty and we cannot. The difficulty half is not a sample-size problem and more calls will not touch it. Until both are handled, a band is a rough grouping, not a verdict, and we would rather publish that than imply a precision the record does not carry.

07How each prediction resolves

Two of these labels carry a known hazard, and the best-known study of expert forecasting names both. Its test for a scoreable claim is the clairvoyance test: a true clairvoyant, handed the prediction, could say whether it came true without asking what was meant^[5]. Our low-falsifiability label is meant for claims that fail that test, though today it also holds past-due claims whose evidence never turned up (§11). The same study found experts explained away misses as off on timing or a close call, and blamed bad luck far more often than they credited good luck^[5]. Our partial label gives credit for exactly those cases, so it is where a lenient judge would be generous. When the judge changed its mind on identical evidence, every change was between partial and wrong, and four of five moved toward the pundit (below).

These labels are not what the formula in §04 runs on. The judge scores each resolved prediction 0–100 on each of the three measures, and the label above is a summary of the direction score rather than an input to it. So the honest way to say what a partial is worth is to measure it. Across the 4404 resolved calls on this site, direction scores average 92 for correct, 65 for partial, 3 for wrong.

One known weakness, published rather than hidden: although the scale runs 0–100, the judge in practice lands on the rubric’s anchor values, using only 16 distinct scores across the whole corpus with 88% of calls on just four of them. A scale that behaves like a short ordinal one throws away information every call, and that is part of why a single call’s spread in §06 is as wide as 32 points. Grading on a finer scale would tighten the score without collecting a single extra prediction.

How stable is a verdict? We have measured it two ways, and the answers differ by a lot.

So the judge’s own randomness is the smaller problem. The larger one is which documents the search happens to return. A single vivid source can carry a verdict, and a different search surfaces a different source. Outside research points to the same weak spot. In one benchmark, the model we use declined to answer 80.5% of the time when given no evidence at all, but only 17.0% of the time when the evidence it was given contradicted itself^[6]. Conflicting sources are exactly what two web searches about the same claim tend to produce. Two changes attack that. The first is half built: since August the grounding judge must copy the exact sentence that settles a claim, and code checks that the sentence really appears in its documents. The check is recorded but does not yet change a verdict, and nearly all scores on this site were built before it existed. The second is the per-document vote above.

The closest published system resolves forecasting questions with three model agents that each search the web, plus a fourth model from another provider that breaks ties. A human checked 100 of its resolutions and found 4 errors, about 95% accuracy^[7]. Two things make that easier than our task. Its questions were written to be resolvable, each with a stated source for the answer. Ours are things people said, extracted after the fact, with no source named in advance. Its design points the same way our measurements do, though: several independent readings, and a different model when they disagree.

08Beating the market

A separate scorecard, for pundits who make market-resolvable calls. Where the PunditScore above asks were they right, this asks a harder question — did they beat the crowd’s own money. It is measured, it is standalone, and it is never folded into the composite.

When a call maps to a real, resolved prediction market, the market hands us free ground truth: the question, the resolution, and the crowd’s implied probability on the day the pundit spoke. The pundit supplies the direction; the market’s price supplies the magnitude. The edge on a single call is the difference of two errors:

o is what happened (1 or 0), m is the crowd’s price at the moment of the call, and p is the pundit — their stated probability if they gave one, or a plain 1/0 for a bare directional call. Positive means they beat the crowd. Calling it when the market sat at 20% and it happened is a large edge; agreeing with a market already at 85% is a small one; being confidently wrong is negative. Nobody is rewarded for picking easy, lopsided markets — there is nothing there to beat.

Each squared error is a Brier score. That rule is strictly proper: a forecaster gets their best expected score only by stating what they actually believe^[2]. The same paper shows that adding a term the forecaster cannot influence keeps a rule proper, and the crowd’s error is such a term, so the edge keeps the property too. That last step is our reading of the paper’s result, not a claim the paper makes about markets. One consequence to know: a bare directional call counts as 100% confidence, so a wrong one costs the most a call can cost.

A pundit’s Market Edge is the average across their resolved market calls, minus 1.28 standard errors: the value we are 90% confident the true average sits above. Fifty is par with the crowd. Below 5 anchored calls we mark it Provisional rather than firm, the same “we don’t know” rule the rest of this page runs on.

That discount needs at least two calls, and most people here have one. A single call has no spread to measure, so its lower bound is simply its edge. On today’s board the highest scores all belong to people with one call each. Read those as one good call, not as a proven edge.

Marking is all it does. We never hide a market call. Every call we hold is shown — on the board, on the profile, in the chart — whether a person made one or thirteen. A resolved market call is settled evidence: the crowd set the price, the market settled the outcome, and no judgement of ours sits in between. One of those is still a fact about the person, and withholding it would show you less than we know. The call-count thresholds elsewhere on this page govern the PunditScore — whether a pundit places against the panel, whether their composite reads as firm. They do not reach this evidence, which is why a pundit with a single market call still has a Market Edge on their profile, labelled as the thin record it is.

What it refuses to do: it counts only calls made before a market settled, on markets that have actually resolved to yes or no, and it credits a taken stance, not professed uncertainty. Before a call counts, a separate model check, one call per quote, confirms that a named person committed to a direction within the market’s timeframe; most candidate quotes fail it. The markets are Polymarket only today. If someone spoke before a market opened, we use its opening price and flag the call. And it never touches the PunditScore. A person can be a middling forecaster overall and still hold a real edge on the narrow set of questions a market priced; we keep those two facts apart rather than blur them into one number.

09Where we say “we don’t know”

A score is only as strong as the resolved evidence behind it, so three different things require three different amounts of it. They are separate on purpose — a chart is a record of what someone said, while a band is a claim about how they compare, and the second needs more evidence than the first.

Many pundits — futurists especially — make calls that simply haven’t come due, and we say so instead of inventing confidence. Those sit in their own section as open calls, outside the score entirely, because nothing has happened yet to judge. A prediction whose horizon has passed can never stay pending: it must resolve or be flagged unresolvable. Forecasting benchmarks avoid guessing the horizon by fixing a resolution date when the question is asked^[8]. Pundits rarely give one, so we read it from what they said, and a misread horizon is one way a call can land in the wrong state. Calls whose horizon is two or more years away are held as open without searching for an outcome.

The mix, so you can see what the score is drawn from: of 4936 claims we have extracted, 4404 have resolved and feed a score, 512 are open, and 20 came due with no resolvable evidence. Keeping open calls out of the score is the right call, but it has a cost we should name — a scored record is the shorter-horizon part of what someone said, and that bites hardest on the people whose calls run longest.

We used to say that pinning makes a score reproducible. It does not, and we have measured why. The formula and its weights are versioned (config v0.2), and model calls run at zero temperature with a fixed seed. But the model is reached by a name its provider can update, and a controlled study in a sister project found that temperature and seed pinning barely reduces a judge’s verdict flips. The larger problem is upstream of the judge: the web changes. Re-running the same searches three days apart returned only 31–38% of the same documents, and §07 shows what different documents do to a verdict. A re-run today would not reproduce every score exactly. What stays fixed is the rule that turns graded claims into a score, and every grade can be traced to its quote and evidence. The archetypes we normalize by: Operational & industry · Macro & structural · Financial & Wall Street · Institutional & advocacy · Investor-banker & operator-prognosticator.

10What a PunditScore is not

It is not an absolute verdict on a person. It reflects the public, checkable record we could find — paywalled research, private notes, and off-the-record calls are not scored, so pundits whose work is mostly private will look thinner than their reputation. Scores are relative and evidence-bounded, calibrated against each other rather than against absolute truth. A score you can’t interrogate would be a failure of the design — so every number here opens to its underlying quote and outcome.

Researchers who build claim-extraction models warn that a model reading what a named person said has no guarantee of matching what they meant. Their answer is to keep a human in the loop and to say when a model did the reading^[9]. Claims arrive faster than any editorial team could check them one by one, so our human in the loop is a public one. Every claim already shows its quote, its source and the evidence behind its outcome. Next, anyone will be able to question a claim or its outcome, and each question will be reviewed by a person (§11). Review that is open to everyone is how we intend to keep the record accurate as it grows.

Three limits we would rather state than have you discover. A small gap is not a gap — at 3.6 points of uncertainty, two adjacent pundits are the same pundit as far as this evidence goes. Nobody here answers the same question — each pundit is scored on the calls they chose to make, so someone taking easy, safe positions can outscore a better forecaster who took hard ones; comparing them relies on the assumption that difficulty evens out, and we are working on anchoring it properly rather than assuming. Of the three, this is the one we think is largest, and it is the reason §06’s 69% is a ceiling. And the judge is a model, which reads evidence and can read it differently on a second pass.

That last one deserves a sharper statement than we gave it before. Where the judge varies at random from call to call, the variation really is inside the noise measured in §06. Where it varies systematically — consistently harsher on hedged language, or on one domain’s vocabulary — it does not cancel out across a record, and it lands in the 6.4 looking like judgment. We have measured the random part; the systematic part needs the same corpus re-scored under a different model, which is the next experiment rather than a claim we can make today.

Published research on model judges sharpens three of these points. First, turning per-claim scores into a ranking of people is a step one survey of the field warns is unreliable: a judge’s separate scores need not agree with its own head-to-head comparisons, and its preferences can be intransitive^[10]. Grading one answer at a time is not itself the weak step: on chat answers, single-answer grading by a strong model agreed with human raters as often as head-to-head comparison did^[11]. The risk is in the conversion to a ranking. We rank anyway, and §06 is our measurement of how far the ranking holds. Second, no published standard says how much agreement makes a judge good enough. The surveys list agreement measures and set no threshold^[10], and they note that judges used to label large datasets, which is our use, have been tested least^[12]. Any bar we adopt is ours to defend. Third, one model family extracts and judges every claim. Judges do favour text that reads as familiar to them^[13], but that was measured on replies the judge wrote itself, and mostly on style rather than facts. Whether it moves our grades is untested.

One more risk runs the other way: the judge may already know the answer. Research on language-model forecasting guards against this by testing only on questions that opened after the model’s training data ends^[14]. We cannot do that, because we score the past. Of the resolved calls on this site, 2,165 of 4,404 were made in 2022 or earlier, and our model’s training data runs to late 2023. We tell the judge to decide from the documents alone. We have not measured whether it does, and a judge grading from memory would look exactly like one grading from evidence whenever the two agree.

11Measured, not yet built

Most of these fixes have been tested on our own data and not yet shipped; the first is planned rather than tested. We list them so the page describes the system that produced today’s scores, not the one we intend to run.

12Sources

We cite a work only after reading it in full, not from its abstract or a summary. Our earlier internal methodology failed that test twice: it cited one paper for a result the paper never measured, and planned a verification step on a method that, read in full, never consults outside evidence. Both are gone.

1. Tang, L., Laban, P. & Durmus, E. (2024). MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents. EMNLP. arxiv.org/abs/2404.10774
2. Gneiting, T. & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378. doi.org/10.1198/016214506000001437
3. Kucinskas, S., Bastani, H. & Karger, E. (2025). ForecastBench: Updated Ranking Methodology. Forecasting Research Institute technical report. www.forecastbench.org
4. Himmelstein, M., Atanasov, P. & Budescu, D. V. (2021). Forecasting forecaster accuracy: Contributions of past performance and individual differences. Judgment and Decision Making, 16(2), 323–362. jbaron.org/journal/vol16.2.html
5. Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press. Chapter 1, “Quantifying the Unquantifiable.” faculty.wharton.upenn.edu/wp-content/uploads/2012/04/Tetlock_2005-EPJ-chapter-1.pdf
6. Zhang, H. & Wu, W. (2026). Do LLMs Know When Evidence is Insufficient? An Evidence Sufficiency Benchmark for Answer-Abstention Calibration in Retrieval-Augmented Generation. Computers, Materials & Continua, 89(1). doi.org/10.32604/cmc.2026.086343
7. Bosse, N. I., Mühlbacher, P., Wildman, J., Phillips, L. & Schwarz, D. (2026). Automating Forecasting Question Generation and Resolution for AI Evaluation. FutureSearch. arxiv.org/abs/2601.22444
8. Karger, E. et al. (2025). ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. ICLR. arxiv.org/abs/2409.19839
9. Ullrich, H., Mlynář, T. & Drchal, J. (2025). Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics. arxiv.org/abs/2502.04955
10. Li, H. et al. (2024). LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arxiv.org/abs/2412.05579
11. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks. arxiv.org/abs/2306.05685
12. Gu, J. et al. (2024). A Survey on LLM-as-a-Judge. arxiv.org/abs/2411.15594
13. Wataoka, K., Takahashi, T. & Ri, R. (2024). Self-Preference Bias in LLM-as-a-Judge. arxiv.org/abs/2410.21819
14. Halawi, D., Zhang, F., Yueh-Han, C. & Steinhardt, J. (2024). Approaching Human-Level Forecasting with Language Models. NeurIPS. arxiv.org/abs/2402.18563