How PunditScore works
Transparency over black-box scoring. Evidence over reputation. Comparability over anecdote.
Every PunditScore traces back to a dated claim, a real-world outcome, and the reasoning in between. Language models extract each claim and judge it against retrieved evidence, one claim per call. Code checks what code can check, numbers are settled by arithmetic where they can be, and a published formula turns the grades into a score. This page describes how that is done today, and what it does and doesn’t claim.
01How a prediction reaches a scorecard
Find what someone predicted, check it is a forecast and theirs, settle what happened, then score it.
A pundit is assessed as three separable questions: did they say it, did it happen, how good was the call. Each can be checked on its own. The rule across all of them is to retrieve first and reason second. Every model call is handed retrieved documents and told to work from them, not from what it remembers.
Each judgment about a claim gets its own model call. An earlier version asked one call to rule on a whole pundit’s claims at once, and it anchored on its first answer and repeated it across claims that had nothing to do with each other.
Search for documents in which the person made predictions: one open search and one limited to older publications, so calls whose due date has already passed are found.
Drop any document that never names the person. Then check the profile against a written identity: what the person does, the employers that anchor them, and the namesakes known to share their name.
Read one document at a time and pull out dated predictions the person made, each with its verbatim quote. Analyst rating tables are parsed by code instead.
Merge near-duplicates, keeping the dated copy, and drop any claim without a date it was made.
Ask of each claim, separately, whether it is a specific, checkable forecast. A second check drops a claim that describes something that had already happened when it was said.
Ask whether the forecast is the person's own, not their organization's plan or a forecast they relay. Then check that the verbatim quote, not our paraphrase of it, makes the prediction.
Review the claim against one written rubric of what counts, shared by every lane. Each verdict names the rule it applied.
A price target or a figure a company files is settled by comparing numbers, before any model reads a document. The comparison is shown in one line beside the claim.
Pin the claim to the period it is about, search reporting from that period only, and judge from those documents. If they do not settle it, the answer is cannot tell.
Grade each resolved claim on three measures against its outcome. The composite is then computed by a fixed formula (§04), not emitted by a model.
Record whether a disclosed stake bears on the claim, and whether it is about the person's own organization. Context sits beside a claim and never changes its grade.
Each step uses a language model chosen for that step, and we change which model fills a step as better ones become available. Every grade records the model that answered, so any grade can be traced to it. Between 25 and 27 October 2026 every panel prediction was re-graded by the dated resolver, so the scores on the site come from one grader. Re-grading is not neutral: on identical code, some outcomes flip between runs, which is why we don’t re-grade to tidy up.
Four ways a prediction comes in
The panel above is one route. The others start from a question or a market rather than a person, and their calls land on the same profiles.
| Lane | How it finds a call | How it is graded | In the PunditScore? |
|---|---|---|---|
| Pundits panel | Search for the person's own published predictions (above). | Arithmetic where numbers settle it, otherwise the dated resolver. | Yes |
| Shared questions | Start from a question with one known public answer, such as a year-end close or an award, and find the people who called it. | Matched against that known answer. | Yes, on the person's profile |
| Market calls | A statement that takes a side on a listed prediction market. | The market's settlement. | For panel pundits, as one resolved prediction. Edge against the crowd is separate (§08) |
| Open calls | A statement on a market still trading, on Polymarket or Kalshi. | Not yet. | No. Shown as open |
Who decided each of the 14,842 resolved predictions on the site:
02Checks run in code
Where a rule can be written as code, it is, and the model’s output is checked or overwritten rather than trusted.
A model can be told a rule and still break it. These run on the panel:
Two judgments stay with a model because no code can make them: whether a sentence is a forecast at all, and what the evidence says happened. Calls from shared questions take a shorter path today. They are checked against their source and graded against a known answer, but they skip the forecast and ownership checks in §01. Bringing them under the same checks is open work.
The quote check compares runs of words, so it catches a quote that is not in the document but not a paraphrase that changes the meaning. Small models now check whether a document supports a sentence about as accurately as the largest models, at a fraction of the cost[1]. They answer does this text say that, which fits a quote. They do not answer did this happen, so they cannot replace the outcome judge.
03What gets measured
Three measures feed the score. Context is published beside it and kept out of it.
Directional, Timing and Revision are each the average of the per-prediction grades across a pundit’s resolved predictions.
Alongside those we publish context: pundit-level characteristics that are deliberately not part of the score. How widely cited someone is, or what commercial stake they hold, tells you how to read a call; neither is evidence that the call was right. Folding them into the number would make one score mean two different things.
04The composite is a formula
The headline score is computed from the three measures by a fixed, versioned weighting. Anyone can re-derive it.
Weights are normalized by analyst type. For Macro & structural analysts, whose calls are due years out, the Timing weight is lowered to 0.15 (from 0.25) and shifted to Directional. A decade-out structural call isn’t judged on timing like a near-term operational one. The analyst types are Operational & industry · Macro & structural · Financial & Wall Street · Institutional & advocacy · Investor-banker & operator-prognosticator.
One rule inside that formula is worth stating rather than leaving you to find. Timing is scored on every resolved prediction, including the ones whose direction was wrong. So a wrong call usually loses points twice: 45% of wrong-direction calls also score zero on Timing. Directional therefore pulls harder on the composite than its 0.50 weight suggests.
One exception to “formula, not a guess”. A pundit with no resolved predictions at all has nothing to average, so the three measures fall back to a model’s estimate from their record. That score is always marked provisional, and it never gets a band or a rank (§09).
05Score bands
A band says where a pundit sits among the people we track, not against a universal standard.
These cuts are not fixed. They are placed at set distances from the mean of the 423 tracked pundits who have at least 8 scored predictions, measured in that group’s own spread (mean 58.2, standard deviation 11.4). “Cohort” here means that whole tracked panel of 423, not other analysts of the same type. The cuts move as the panel grows. Cut points and band edges are rounded to one decimal place; the comparison itself runs on the unrounded values. This spread reads 11.4 here and 11.4 in §06 because §06 measures the per-call scores rather than the published composites.
The middle band is the widest of the five, but widest is not the same as most: on today’s panel about 67% of banded pundits get a label that says above or below, and only 138 of 423 sit at the average. The labels are only worth as much as §06 says they are: at 59% agreement a five-way split is supportable, where at the agreement we were reporting before this corpus was collected it was not. If that number falls again, the bands should get coarser rather than the caveat longer. A pundit with fewer than 8 scored predictions gets no band and no rank: too few scored calls cannot tell a poor forecaster from an unlucky one.
06How much of the number is real
If one pundit shows 6 points above another, about 58% of that gap is a real difference in judgment, and that figure is a ceiling.
A composite is an average, so the spread between pundits is part skill and part luck, and the two can be measured apart. Across the 423 ranked pundits with at least 8 scored calls:
Read that plainly: if one pundit shows 6 points above another, expect about 58% of that gap, near 3.5 points, to be a real difference in judgment, and the rest to be the luck of which calls they happened to make. That 58% is a share of variance, which is the figure that tells you how far to discount a gap. Compared as spreads rather than gaps the same data reads 76%, because 8.7 is 76% of 11.4. Both numbers are true and they answer different questions.
Sampling noise across the panel has a spread of 7.4 points. With skill, it adds up to the 11.4 we display. A single pundit’s own score carries an uncertainty of about 6.9 points at the median record length of 20 resolved calls, more for a thinner record and less for a fuller one. Either way, two people a few points apart are not distinguishable. Split a pundit’s own record in half and the halves agree on the ranking at 0.42 out of 1; stepped up to full record length that is roughly 0.59, which is the number comparable with the 58% above. Two independent routes give the same answer.
One caveat on the 8.7, and it cuts against us. That figure is what persists across a pundit’s record once luck is subtracted, which is not quite the same as skill. A steady habit of making easy, safe calls persists exactly as well as good judgment does, so it survives the subtraction and gets counted as judgment. The real skill spread is therefore somewhere below 8.7, and the 58% is a ceiling rather than a floor.
The standard reference on scoring forecasts says it directly: scores compare cleanly only when they cover exactly the same forecasting situations, and otherwise how predictable the questions were gets tangled with how good the forecaster was[2]. Forecasting benchmarks hit the same wall when forecasters answer different questions. Their fix is to estimate how hard each question was from everyone who answered it, and then credit each forecaster for beating that difficulty[3]. The method needs questions shared across forecasters. Most of the panel’s record still has none: each pundit is graded on what they chose to say, and a difficulty model fitted to it simply hands back each pundit’s raw average. Since November 2026 part of the record comes from shared questions, where many people called the same outcome, and that is what the method needs. A first score against peers on those questions does not yet separate people reliably, so it is not shown (§11).
The sampling half of this is a sample-size problem and more resolved calls fix it: the median scored pundit has 20 resolved calls, and it takes about 51 before four fifths of their score is signal. That is the same order of record a forecasting tournament needed: there, past accuracy became the best guide to future accuracy once about 20 questions had resolved, and kept adding information through 80[4]. Everyone in that tournament answered from one shared pool of questions, so it could adjust for difficulty. The difficulty half is not a sample-size problem, and more calls of the kind we have will not touch it. Until both are handled, a band is a rough grouping, not a verdict.
07How each prediction resolves
Each prediction ends in one of five states. The judge’s weak spot has been the documents it reads, more than how it reads them.
Two of these labels carry a known hazard, and the best-known study of expert forecasting names both. Its test for a scoreable claim is the clairvoyance test: a true clairvoyant, handed the prediction, could say whether it came true without asking what was meant[5]. A claim that fails that test is too vague. Most claims with no score pass it: they are past due, we looked for what happened, and the reporting did not say, so they have no verdict. The same study found experts explained away misses as off on timing or a close call, and blamed bad luck far more often than they credited good luck[5]. Our partial label gives credit for exactly those cases, so it is where the judge’s reading matters most.
These labels are not what the formula in §04 runs on. The grader scores each resolved prediction 0–100 on each of the three measures, and the label is a summary of the direction score rather than an input to it. So the honest way to say what a partial is worth is to measure it. Across the 14,842 resolved calls on this site, direction scores average 90 for correct, 59 for partial, 5 for wrong.
One known weakness: although the scale runs 0–100, the grader in practice lands on the rubric’s anchor values, using only 18 distinct scores across the whole corpus with 84% of calls on just four of them. A scale that behaves like a short ordinal one throws away information every call, and that is part of why a single call’s spread in §06 is as wide as 31 points.
How the dated resolver works
A stored claim often carries no period of its own: “rates will go up by the end of the year”. Searching for the bare claim returns whatever year the search engine likes, and an earlier grader graded a 2010 tennis pick on the 2012 Olympics. The dated resolver makes three calls per claim. It first restates the claim with absolute dates and names its deadline. It then searches reporting published in that period only. Last, it judges from facts inside that period, and answers cannot tell when they do not settle it, which leaves the claim with no verdict. A deadline still ahead means pending, with no search. A grade that quotes a projection rather than a report of what happened is refused.
What we measured about the earlier grader
We tested the grader that came before the dated resolver, and what we found is why it was replaced. Reading the same documents twice, it sometimes changed its verdict, always between partial and wrong and mostly toward the pundit. Judging the same claim against two different sets of documents about the same event, it agreed with itself only about half the time, and not much more often even when it said both times that its documents settled the question.
Those tests measure agreement, not correctness. A judge that reads five documents the same wrong way is stably wrong, and none of them would catch it. Checking correctness needs claims whose answer is known independently of any judge, such as those settled by arithmetic on share prices or company filings. Building that answer key turned up faults in the key itself: claims checked over a single trading day because a ratings table’s date column meant “the analyst filed this today”, twelve-month targets graded wrong before their deadline, prices from before a claim counted toward it, and a target already met when it was set counted as reached. Each is fixed.
On the answer key that remained, the earlier grader was wrong more often than right, and its errors ran the same way: a correct call marked wrong, such as a share-price target the stock went on to pass with room to spare, or a revenue forecast the company’s filing matched. A person grading the same claims without seeing the judge’s verdict made the same kind of error. What misled both was in the documents: a share price quoted after a ten-for-one split and compared against a pre-split target, and articles about analysts cutting a target read as the company never reaching it. That is why price targets and reported figures are now settled by arithmetic before any judge reads a document, and why the resolver reads only the claim’s own period.
How a price target is settled. A target with no stated deadline is read as a twelve-month target. It counts as reached if the stock closes at or above it on any trading day in the window, and as missed only once the window has closed. That is the “met at some time” measure in the largest study of analysts’ targets[15], which found 64% of 492,647 twelve-month targets reached at some time and 38% reached at the end. Our closed windows land in the same range, and in their data and in ours the rate falls as the target sits further above the price. The two samples cover different years, so this shows our arithmetic behaves like theirs, not that either rate is the right one. A target more than four times the price when it was set is not settled at all: the study drops those as data errors, and the ones we found were another company’s shares.
Which documents a search returns has been the larger problem than the judge’s own randomness. In one benchmark, the model we used then declined to answer 80.5% of the time when given no evidence at all, but only 17.0% of the time when the evidence it was given contradicted itself[6]. Conflicting sources are exactly what two web searches about the same claim tend to produce. The closest published system resolves forecasting questions with three model agents that each search the web, plus a fourth model from another provider that breaks ties. A human checked 100 of its resolutions and found 4 errors[7]. Its questions were written to be resolvable, each with a stated source for the answer. Ours are things people said, extracted after the fact. Its design points the same way our measurements do: several independent readings, and a different model when they disagree (§11).
08Beating the market
A separate scorecard: did they beat the crowd’s own money? It is never folded into the PunditScore.
When a call maps to a real, resolved prediction market, the market hands us ground truth: the question, the resolution, and the crowd’s implied probability on the day the person spoke. The person supplies the direction; the market’s price supplies the magnitude. The edge on a single call is the difference of two errors:
o is what happened (1 or 0), m is the crowd’s price at the moment of the call, and p is the person: their stated probability if they gave one, or a plain 1/0 for a bare directional call. Positive means they beat the crowd. Calling it when the market sat at 20% and it happened is a large edge; agreeing with a market already at 85% is a small one; being confidently wrong is negative. Nobody is rewarded for picking lopsided markets, because there is nothing there to beat.
Each squared error is a Brier score. That rule is strictly proper: a forecaster gets their best expected score only by stating what they actually believe[2]. The same paper shows that adding a term the forecaster cannot influence keeps a rule proper, and the crowd’s error is such a term, so the edge keeps the property too. That last step is our reading of the paper’s result, not a claim the paper makes about markets. A bare directional call counts as 100% confidence, so a wrong one costs the most a call can cost.
A person’s Market Edge is the average across their resolved market calls, minus 1.28 standard errors: the value we are 90% confident the true average sits above. Fifty is par with the crowd. Below 5 calls we mark it Provisional rather than firm. That discount needs at least two calls, and most people on the board have one: a single call has no spread to measure, so its lower bound is simply its edge. Read a high score on one call as one good call, not as a proven edge.
Marking is all it does. We never hide a market call. Every call we hold is shown, on the board, on the profile and in the chart, whether a person made one call or many. A resolved market call is settled evidence: the crowd set the price, the market settled the outcome, and no judgement of ours sits in between. The call-count thresholds elsewhere on this page govern the PunditScore. They do not reach this evidence.
Where the two meet. 296 of the board’s 361 forecasters are also on the Pundits panel. For them, each settled market call also counts as one resolved prediction in their PunditScore, graded right or wrong by the settlement. Market Edge itself is not a term in the PunditScore.
What it refuses to do: it counts only calls made before a market settled, on markets that have actually resolved to yes or no, and it credits a taken stance, not professed uncertainty. Three checks run with no model involved: the quote must appear in the document it was read in, the document must name the person, and the date must be one the document supports. A model check, one call per quote, then confirms that a named person committed to a direction within the market’s timeframe; most candidate quotes fail it. Two more run after it, each with the passage around the quote in view: that the named person is the one speaking, and that the statement is about this market’s question and period. A statement made more than a year before its market opened is not counted. If someone spoke before a market opened, we use its opening price and flag the call. Settled calls are on Polymarket today. Open calls on markets still trading also come from Kalshi, and are shown but not scored.
A person’s statement about their own side is shown and marked, and scored one way only. That covers what they themselves will do (a president on his own tariffs, a chief executive on his own company’s launch), and what the government, agency or company they hold a post in will do or is arguing for. Being right there mostly shows they kept their word, so it earns nothing. Being wrong about something they control or sit beside is informative, so it counts in full.
What the crowd did next. Each call also carries what the market’s price did after it: how far it had moved toward the side the person took 7 and 30 days later, and whether it first moved 15 points their way or 15 points against them. Day 0 is the day of the price the call was scored at, which for a call made before the market opened is the market’s first price. A move counts only if it came before the outcome was known: the clock stops the day before the price first came within 2 points of 0% or 100%, or the day before the market settled, whichever was first. Three kinds of call are drawn but not timed: those on a market already near 0% or 100% (21), those by someone who decides the event or sits inside the body that does (25), and those whose outcome was known by the next day (25). This describes a call; it is not a measure of skill. Of the 478 timed calls, the crowd moved 15 points toward the call first 161 times, against it first 132 times, and neither way 185 times.
09Where we say “we don’t know”
A score is only as strong as the resolved evidence behind it, and we say so where it is thin.
Many pundits, futurists especially, make calls that haven’t come due, and we say so instead of inventing confidence. Those sit in their own section as open calls, outside the score. A call with a stated deadline still ahead is held open without searching for an outcome. A call with no stated deadline gets two years from the day it was said: inside them it can come true early but cannot be marked wrong. Forecasting benchmarks avoid guessing the due date by fixing a resolution date when the question is asked[8]. Pundits rarely give one, so we read it from what they said, and a misread due date is one way a call can land in the wrong state.
The mix, so you can see what the score is drawn from: of 24,623 claims we have extracted, 14,842 have resolved and feed a score, 5,190 are open, and 4,027 came due with no verdict. Keeping open calls out of the score is right, but it has a cost: a scored record is the sooner-due part of what someone said, and that bites hardest on the people whose calls run longest.
A re-run would not reproduce every score exactly, and we have measured why. The formula and its weights are versioned (config v0.3). Where a model allows it, we run it at zero temperature with a fixed seed. Some models accept neither setting, and a controlled study in a sister project found that such pinning barely reduces a judge’s verdict flips anyway. The larger problem is upstream of the judge: the web changes. Re-running the same searches a few days apart returned mostly different documents. What stays fixed is the rule that turns graded claims into a score, and every grade can be traced to its quote, its evidence and the grader that made it.
10What a PunditScore is not
Not an absolute verdict on a person: a relative score on the public, checkable record we could find.
Paywalled research, private notes and off-the-record calls are not scored, so pundits whose work is mostly private will look thinner than their reputation. Scores are relative and evidence-bounded, calibrated against each other rather than against absolute truth. Every number opens to its underlying quote and outcome.
Researchers who build claim-extraction models warn that a model reading what a named person said has no guarantee of matching what they meant. Their answer is to keep a human in the loop and to say when a model did the reading[9]. Claims arrive faster than any editorial team could check them one by one, so our human in the loop is a public one. Every claim shows its quote, its source and the evidence behind its outcome, and every claim carries a flag. A signed-in reader can say which part looks wrong (the quote, the person it is attributed to, a date, or how it was resolved) and add a comment. A person at PunditScore reviews each flag, and an upheld flag corrects the record, or moves the claim to the person who actually made it.
Three limits we would rather state than have you discover. A small gap is not a gap: at 6.9 points of uncertainty, two adjacent pundits are the same pundit as far as this evidence goes. Most of the panel does not answer the same questions: each pundit is scored on the calls they chose to make, so someone taking easy, safe positions can outscore a better forecaster who took hard ones. Of the three, this is the one we think is largest, and it is the reason §06’s 58% is a ceiling. And the judge is a model, which reads evidence and can read it differently on a second pass.
Where the judge varies at random from call to call, the variation is inside the noise measured in §06. Where it varies systematically, consistently harsher on hedged language or on one domain’s vocabulary, it does not cancel out across a record, and it lands in the 8.7 looking like judgment. Measuring that needs the same corpus re-graded by a different model, which we have not done.
Published research on model judges sharpens three of these points. First, turning per-claim scores into a ranking of people is a step one survey of the field warns is unreliable: a judge’s separate scores need not agree with its own head-to-head comparisons, and its preferences can be intransitive[10]. Grading one answer at a time is not itself the weak step: on chat answers, single-answer grading by a strong model agreed with human raters as often as head-to-head comparison did[11]. The risk is in the conversion to a ranking, and §06 is our measurement of how far the ranking holds. Second, no published standard says how much agreement makes a judge good enough. The surveys list agreement measures and set no threshold[10], and they note that judges used to label large datasets, which is our use, have been tested least[12]. Any bar we adopt is ours to defend. Third, models from one provider extract and judge every claim. Judges do favour text that reads as familiar to them[13], but that was measured on replies the judge wrote itself, and mostly on style rather than facts. Whether it moves our grades is untested.
One more risk runs the other way: the judge may already know the answer. Research on language-model forecasting guards against this by testing only on questions that opened after the model’s training data ends[14]. We cannot do that, because we score the past. Of the resolved calls on this site, 9,875 of 14,842 were made in 2022 or earlier, before the training data of the models we use ends. We tell the judge to decide from the documents alone, and the dated resolver gives it only the claim’s own period to read. We have not measured whether it does, and a judge grading from memory would look exactly like one grading from evidence whenever the two agree.
11Tried, and still open
What we have tested or planned, and where each stands. The page above describes the system that made today’s scores.
12Sources
We cite a work only after reading it in full, not from its abstract or a summary.
Our earlier internal methodology failed that test twice: it cited one paper for a result the paper never measured, and planned a verification step on a method that, read in full, never consults outside evidence. Both are gone.
- Tang, L., Laban, P. & Durmus, E. (2024). MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents. EMNLP. arxiv.org/abs/2404.10774
- Gneiting, T. & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378. doi.org/10.1198/016214506000001437
- Kucinskas, S., Bastani, H. & Karger, E. (2025). ForecastBench: Updated Ranking Methodology. Forecasting Research Institute technical report. www.forecastbench.org
- Himmelstein, M., Atanasov, P. & Budescu, D. V. (2021). Forecasting forecaster accuracy: Contributions of past performance and individual differences. Judgment and Decision Making, 16(2), 323–362. jbaron.org/journal/vol16.2.html
- Tetlock, P. E. (2005). Expert Political Judgment: How Good Is It? How Can We Know? Princeton University Press. Chapter 1, “Quantifying the Unquantifiable.” faculty.wharton.upenn.edu/wp-content/uploads/2012/04/Tetlock_2005-EPJ-chapter-1.pdf
- Zhang, H. & Wu, W. (2026). Do LLMs Know When Evidence is Insufficient? An Evidence Sufficiency Benchmark for Answer-Abstention Calibration in Retrieval-Augmented Generation. Computers, Materials & Continua, 89(1). doi.org/10.32604/cmc.2026.086343
- Bosse, N. I., Mühlbacher, P., Wildman, J., Phillips, L. & Schwarz, D. (2026). Automating Forecasting Question Generation and Resolution for AI Evaluation. FutureSearch. arxiv.org/abs/2601.22444
- Karger, E. et al. (2025). ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities. ICLR. arxiv.org/abs/2409.19839
- Ullrich, H., Mlynář, T. & Drchal, J. (2025). Claim Extraction for Fact-Checking: Data, Models, and Automated Metrics. arxiv.org/abs/2502.04955
- Li, H. et al. (2024). LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arxiv.org/abs/2412.05579
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. NeurIPS Datasets and Benchmarks. arxiv.org/abs/2306.05685
- Gu, J. et al. (2024). A Survey on LLM-as-a-Judge. arxiv.org/abs/2411.15594
- Wataoka, K., Takahashi, T. & Ri, R. (2024). Self-Preference Bias in LLM-as-a-Judge. arxiv.org/abs/2410.21819
- Halawi, D., Zhang, F., Yueh-Han, C. & Steinhardt, J. (2024). Approaching Human-Level Forecasting with Language Models. NeurIPS. arxiv.org/abs/2402.18563
- Bradshaw, M. T., Brown, L. D. & Huang, K. (2013). Do sell-side analysts exhibit differential target price forecasting ability? Review of Accounting Studies, 18(4). doi.org/10.1007/s11142-012-9216-5