how early AI reached IMO gold-medal level
In 2022, domain experts expected it by 2030 and superforecasters by 2035. It happened in July 2025.
Forecasting Research Institute (FRI), 2026 [6]Home / Forecasting evidence
Evidence base · November 2026Foresight explores many futures; forecasting puts numbers on a few of them. Policy needs both. This page collects the track record: which forecasters are well calibrated, which institutions err in a predictable direction, and what that means for anyone who plans with official projections.
In 2022, domain experts expected it by 2030 and superforecasters by 2035. It happened in July 2025.
Forecasting Research Institute (FRI), 2026 [6]2022 forecasts for 2024: 0.32 to 0.60 MtCO₂ a year. Projected actual: 0.01 Mt. Hardware-bound tech moved far slower than expected.
FRI, XPT near-term accuracy [47]Across 123 countries (1992–2020): export and growth forecasts systematically optimistic, reserve forecasts systematically pessimistic.
Eicher and Kawai [69]They beat prediction markets (Brier 0.210 vs 0.227) in the IARPA Good Judgment Project, across 261 geopolitical events.
Atanasov et al., Management Science, 2017 [7]The CDC and Reich Lab multi-model ensembles also beat 84–91% of trend and complex baselines, across more than 2.5 million audited US forecasts.
Hubverse evaluation [22]Metaculus Pro Forecasters beat the best AI bots on 99 matched questions in Spring 2026, although a bot won the Summer 2026 Metaculus Cup outright.
Metaculus, AI forecasting in 2026 [13]Public-sector projections steer trillions in budgets and investment. Their errors are not just random shocks: audits find biases that run in a consistent direction, often driven by incentives or by assumptions baked into the model.
| Institution and product | Scope | How it is audited | Documented bias |
|---|---|---|---|
| Philadelphia Fed Survey of Professional Forecasters (since 1968; Philly Fed since 1990) | Oldest US quarterly panel: GDP, CPI and PCE inflation, unemployment, density ranges | Individual, ID-tracked errors published online | Density forecasts historically overestimated the probability of large inflation shocks; inflation surprises are serially correlated [66][67] |
| ECB Survey of Professional Forecasters (since 1999) | About 75 euro-area forecasters, around 59 respondents a round | Mean error, RMSE, probability distributions | Limited inflation bias after adjusting for shocks, but consistent GDP overprediction since the 2001 slowdown, sharpest at the two-year horizon [68] |
| IMF World Economic Outlook and crisis programmes | Global surveillance; 602 lending programmes in 123 countries | Holden-Peel and Mincer-Zarnowitz tests; IEO (2014); Eicher and Kawai (2021) | WEO unbiased for advanced economies, but crisis forecasts optimistic on exports and growth; reserve coverage averaged 55% below actuals in the worst balance-of-payments quartile because known conditionality effects were left out [69][70][71] |
| IEA World Energy Outlook and US EIA Annual Energy Outlook | Energy demand, fuel mix, transition scenarios (WEO 2026 published 19 November 2026) | Projected versus realised capacity additions (Hoekstra audits, 2002–2026) | Multi-decade under-forecasting of solar PV. WEO 2004 saw renewables' share of electricity rising only from 18% to 19% by 2030; global solar share grew ninefold from 0.15% (2010) to 1.3% (2016) [73][74][77][78] |
| US CDC and Reich Lab: COVID-19 Forecast Hub, FluSight, Hubverse | About two dozen disease hubs; more than 120 COVID teams over 160 weeks | Weighted interval score, relative WIS, 95% coverage | The counter-example: open, standardised multi-model ensembles beat almost all baselines; FluSight four-week skill now exceeds its first season's one-week skill. Ensembles still struggle at sudden turning points [22][79][81] |
The pattern matters for foresight practice. Scenario baselines built on current or stated policies systematically flatten exponential-to-logistic adoption curves, and growth forecasts made inside an institution that must defend a programme lean optimistic. Treat an official projection as one perspective, with a known direction of error, not as the probable future of the futures cone.
Challenge baseline assumptions in Module 3: assumption premortemThe strongest finding in forecasting research is that how you combine judgements matters as much as whose judgements you collect.
Four seasons, 100–150 geopolitical questions a year, more than 2,800 volunteers. The top 2% each season became “superforecasters”. The widely quoted 30% edge over intelligence analysts came mostly from the aggregation algorithm (weighting and extremising), not from the play-money market, which only matched the analysts' own market [7][17][107].
Tetlock's Expert Political Judgment (2005) found that eclectic “foxes” beat big-idea “hedgehogs” on calibration and discrimination, simple statistical extrapolation beat the average expert, and media prominence was inversely related to accuracy [86].
On ForecastBench, superforecasters scored a Brier of 0.081 in 2024–25 and a Brier Index of 70.2 in May 2026. Their edge rests on probabilistic literacy, reference classes, small frequent updates and team extremising; roughly 70% stay in the top tier year on year [19][49].
Dollar volume on the four largest venues reached $137.5 billion in January–October 2026, more than six times the $21.9 billion traded in all of 2025, mostly on sports and crypto rather than politics. A Kalshi study of 2.24 million markets found Brier scores falling from about 0.08–0.09 three months out to about 0.02 at close [1][8].
Concrete, timestamped predictions with known outcomes are the only honest test. The Forecasting Research Institute's audits show a striking split by domain.
| Question | Elicited | Forecast | Outcome | Verdict |
|---|---|---|---|---|
| Year AI reaches IMO gold-medal level | 2022 (XPT) | 2030 experts / 2035 superforecasters | July 2025 | Underestimate |
| AI matches the baseline on a biorisk troubleshooting test (VCT) | Nov 2024–Feb 2025 | 2030 experts / 2034 superforecasters | April 2025 | Underestimate |
| Cost of the largest AI training run by 2030 | 2022 (XPT) | $180M experts / $100M superforecasters | $388M already by mid-2025 | Underestimate |
| Highest annual recurring revenue of an AI company, end 2026 | FRI economic study | $16–25B | About $100B (Sept 2026) | 4–6× under |
| Share of US work hours assisted by generative AI, 2025 | Sept–Oct 2025 (LEAP) | 4.0% experts / 3.6% superforecasters | 5.7% | Moderate under |
| Top Epoch Capabilities Index score, end 2026 | 2025–26 | 165 experts / 169 superforecasters | 167 | Accurate |
| Autonomous share of US ride-hailing trips, 2027 | LEAP panel | 7.3% experts / 2.0% superforecasters | About 2.5% (tracking) | Experts over |
| Direct air capture and storage, 2024 | 2022 (XPT) | 0.60 Mt experts / 0.32 Mt superforecasters | 0.01 Mt (projected) | 32–60× over |
Sources: FRI, “How accurate have AI progress forecasts been so far?” [6] and the XPT near-term accuracy report [47]. Some 2026 outcomes are tracking estimates, not final resolutions.
Two lessons. First, near-term accuracy is not a reliable proxy for long-term credibility: on the 38 near-term XPT questions resolved by mid-2025, superforecasters and domain experts were equally accurate, and both shared the same blind spots [47]. Second, the errors are not uniformly pessimistic. Software and revenue curves compounded faster than anyone expected, while physical-world deployment, from carbon removal to driverless taxis, met friction that rewarded outside-view scepticism. That boundary is a useful rule of thumb when you build scenario axes.
In two years AI forecasting went from trailing the public to winning open tournaments. How you read that depends on what is being measured.
A bot won the Summer 2026 Metaculus Cup (5 September 2026); bots took 1st, 2nd and 5th. Cups reward 24/7 coverage, instant news and frequent updating as well as accuracy [3].
On identical questions in controlled windows, Metaculus Pros beat the best bot teams in five straight comparisons, most recently by 1.25 points per question. Their edge is mainly discrimination, not calibration [13].
Metaculus's survey of 39 bot builders shows what separates winning pipelines: a frontier model backbone; capping probabilities away from 0 and 100% (r = +0.48); two or more independent research feeds (r = +0.42); explicit base rates and lookups of similar resolved questions; around 28 model calls per question rather than 7; and Platt-scaling recalibration [13]. These are, almost one for one, the habits of good human forecasters.
Where resolution lies decades away, disagreement between groups is itself the signal.
| Topic | Group | Vintage | Median estimate | Trend |
|---|---|---|---|---|
| General AI arrival | Metaculus community | Oct 2026 | January 2031 | Compressed sharply since 2022 |
| Human-level machine intelligence | AI Impacts survey (n = 1,580) | Dec 2024 | 2042 | 2061 → 2059 → 2047 → 2042 across four waves |
| AGI (strict definition) | FRI superforecasters / experts | Mid-2026 | 2047 / 2050 | Pulled in from about 2060 in 2022 |
| AI catastrophe killing >10% of humanity by 2100 | FRI experts / superforecasters | Mid-2026 | 5% / 2.4% | Superforecasters at about half the expert view |
| Societal impact of AI by 2040 | FRI LEAP panel | 2025 → 2026 | “Technology of the century” (electricity-scale) | Rising weight on “technology of the millennium” |
| Nuclear conflict with more deaths than WWII within 25 years | FRI nuclear risk panel | 2022–2024 | 5% | Non-AI tail-risk anchor |
Respondents are sometimes internally inconsistent on long tails, for example rating transformative growth by 2070 as likelier than strong growth by 2100 [64]. Use these numbers as comparative belief distributions and weigh the direction of updates across methods (here: timelines compressing in surveys, platforms and compute models at once) more than any single median [63][65][96][112].
Put these signals in context on Horizon 2026–2050If a unit cannot score its own forecasts, it cannot learn from them. Five rules cover almost every case.
| Rule | Use it for | Watch out for |
|---|---|---|
| Brier score | Binary questions. Mean squared error of the probability; 0 is perfect, a constant 50% scores 0.25 | Legacy datasets (Good Judgment Project, Good Judgment Open) use a 0–2 scale: divide by 2 before comparing [7] |
| Murphy decomposition | Splitting a Brier score into reliability (calibration), resolution (discrimination) and uncertainty (question difficulty) | Uncertainty is outside the forecaster's control, so raw scores on easy questions flatter [8] |
| Brier skill score / Brier Index | Comparing across question sets: BSS = 1 − BS / BSreference; ForecastBench's index adjusts for difficulty (higher is better) | The reference forecast must be stated [12] |
| Log and peer scores | Penalising overconfident misses harshly; Metaculus's time-averaged scores reward early, frequent updating | Not comparable with Brier values [11][21] |
| CRPS and weighted interval score | Continuous and quantile forecasts, as in epidemic hubs | Needs full distributions, not point estimates [114] |
Two practical controls: record when the outcome was actually decided, not only when the question formally closed (14% of markets in one 2026 study were settled more than a day before close [2]), and score at fixed offsets such as 30 days, 7 days and 1 day before the decision. A free tool such as Fatebook is enough to start a team track record [43].
Compiled from the research dossier Global Forecasting Ecosystem Dataset (November 2026). Numbers keep the source's original reference numbers. Exchange-authored and vendor figures are labelled as such in the text; raw Brier scores are not directly comparable across venues.
Run your policy through four structurally different futures, then decide which signposts to score.