Within Calibration
How To Make A Forecast Actually Checkable
A forecast becomes useful when the metric, deadline, threshold and data source are clear before the result is known.
On this page
- Why vague forecasts cannot teach you much
- The four parts of a good resolution rule
- Examples from deadlines, costs and campaign targets
Page outline Jump by section
Introduction
A forecast only becomes useful for learning when everyone agrees, before the outcome is known, how it will be judged. Without a clear resolution rule, hindsight can quietly change the meaning of a prediction. A statement such as “the project will probably succeed” can always be reinterpreted after events unfold. By contrast, “there is a 70% chance the project will launch to paying customers by 30 September 2027, according to the company’s public release” can be checked objectively.
Resolution rules are therefore a practical tool for improving thinking and confidence calibration. They force you to specify exactly what counts as success or failure before you know the answer, reducing motivated reasoning, memory distortion and endless arguments about whether a forecast was “basically right”. Competitive forecasting platforms such as Metaculus and Good Judgment treat clear resolution criteria as essential because ambiguous questions cannot produce reliable feedback. [metaculus.com+2metaculus.com]metaculus.comQuestion writing and submission guidelinesSpecifying precise resolution criteria. Resolution criteria are the backbone of any forecasting…
Why vague forecasts cannot teach you much
Learning from predictions depends on comparing what you expected with what actually happened. That comparison breaks down when the forecast itself is unclear.
Consider these examples:
- “Sales will increase significantly.”
- “The software launch will be a success.”
- “Inflation will come down soon.”
Each statement leaves multiple questions unanswered. What counts as “significantly”? Which software release? How much lower is “come down”? When is “soon”?
Because these terms have no agreed meaning, people can unconsciously redefine them after the fact. If sales rise by 3%, one person may call that significant while another expected 20%. Both can sincerely believe they remembered the original prediction correctly.
This is one reason forecasting communities devote considerable effort to writing questions before anyone begins estimating probabilities. If forecasters disagree about what is being predicted, differences in accuracy become impossible to interpret. Metaculus explicitly recommends “tight” resolution criteria that leave as little room as possible for later discretion or disagreement. [metaculus.com]metaculus.comQuestion writing and submission guidelinesSpecifying precise resolution criteria. Resolution criteria are the backbone of any forecasting…
The four parts of a good resolution rule
A checkable forecast normally specifies four elements before any evidence arrives.
1. The metric
The outcome must be measurable.
Instead of:
“Customer satisfaction will improve.”
Use:
“Average customer satisfaction score will exceed 4.3 out of 5.”
The metric identifies exactly what quantity matters.
2. The deadline
Every forecast needs a point at which it becomes resolvable.
Instead of:
“The company will become profitable.”
Use:
“The company will report positive net income for the financial year ending 31 December 2027.”
Without a deadline, almost any prediction can survive indefinitely by claiming it simply has not happened yet.
3. The threshold
Many outcomes vary continuously rather than occurring as simple yes-or-no events. The threshold specifies where success begins.
Examples include:
- Revenue exceeds £10 million.
- Construction is at least 90% complete.
- Polling average reaches 50%.
- Inflation falls below 2%.
Explicit thresholds prevent disputes over borderline cases.
4. The resolution source
Someone must know where the answer will come from.
Possible sources include:
- an official government statistical release
- audited company accounts
- an election authority
- a named public database
- a contractually defined project report
Forecasting platforms encourage naming the primary source and, where appropriate, a fallback source in case the original data become unavailable. Otherwise a prediction can become impossible to resolve if the chosen publication disappears or changes methodology. [metaculus.com+2Rethink Priorities]metaculus.comMetaculus Question Approval ChecklistWhen naming a resolution source, are we trying to forecast that particular source more than the "tru…
Why the data source matters as much as the prediction
Many disagreements arise not because people disagree about the future, but because they unknowingly rely on different measurements.
Suppose someone predicts:
“The unemployment rate will fall below 4%.”
Which unemployment rate?
- National statistical office?
- International Labour Organization estimate?
- A private survey?
- Monthly or quarterly figure?
Each source may produce different numbers.
Good resolution rules identify the exact dataset in advance. Where possible they also specify what happens if revisions occur later, whether preliminary or final figures are used, and how ties or rounding are handled. Forecast-writing guidance from Metaculus repeatedly emphasises defining terms carefully and minimising opportunities for ex-post disputes. [metaculus.com]metaculus.comQuestion writing and submission guidelinesSpecifying precise resolution criteria. Resolution criteria are the backbone of any forecasting…
Examples from deadlines, costs and campaign targets
The difference between vague and checkable forecasts becomes obvious when rewritten.
Vague forecastCheckable forecast”The project will finish on time.”“There is a 75% chance the project reaches practical completion by 31 March 2028 according to the signed completion certificate.”“Construction will stay within budget.”“Final approved construction cost will not exceed £24 million excluding VAT, according to the audited project accounts.”“The campaign will perform well.”“The campaign will generate at least 4,000 qualified leads by 30 June as recorded in the CRM system.”“The product launch will succeed.”“The product will record at least 10,000 paying customers within 90 days of launch according to company financial reporting.”
Notice that each version identifies:
- what is measured
- the pass-fail threshold
- the deadline
- the source used to judge the outcome
None requires later interpretation.
Planning for awkward cases before they happen
Real-world events rarely unfold exactly as expected. Good resolution rules anticipate unusual situations instead of improvising later.
Examples include:
- Data revisions. Will the first published figure count, or the final revised figure?
- Cancelled events. If a product launch is cancelled, does the forecast automatically resolve as “no”?
- Merged organisations. If a company changes its legal structure, which entity counts?
- Missing publications. What happens if the planned statistical release is delayed indefinitely?
Experienced forecasting platforms recommend considering these edge cases before the question opens. A small amount of extra precision at the start avoids much larger disagreements after the outcome becomes important. [Rethink Priorities+2metaculus.com]rethinkpriorities.orgtypes of specification problems in forecastingRethink PrioritiesTypes of specification problems in forecasting20 Jul 2021 — Metaculus has a guide on how to write unambiguous and usefu…
Common mistakes that weaken resolution rules
Several recurring problems make forecasts difficult to evaluate.
Using subjective language. Words such as “successful”, “major”, “significant”, “soon” and “strong” often hide multiple interpretations.
Changing definitions midstream. Once new information appears, there is a temptation to redefine success. This undermines any attempt to measure forecasting skill.
Leaving out the deadline. Predictions without an end date cannot clearly fail.
Choosing unstable sources. If a prediction depends on an unpublished spreadsheet or an unofficial website, later verification may become impossible.
Predicting a source instead of the underlying event unintentionally. Sometimes the objective is to forecast reality rather than whether one particular organisation reports it. Where this distinction matters, forecasting guidelines recommend clarifying whether fallback sources should be used if the primary source proves unsuitable. [metaculus.com]metaculus.comMetaculus Question Approval ChecklistWhen naming a resolution source, are we trying to forecast that particular source more than the "tru…
Resolution rules improve calibration, not just record-keeping
The immediate benefit of a resolution rule is that it settles whether a forecast was correct. The deeper benefit is that it creates honest feedback.
When forecasters repeatedly make predictions with clear resolution criteria, they can compare stated probabilities with actual outcomes using measures such as calibration curves or scoring rules. If predictions assigned 70% probability occur roughly 70% of the time, confidence is well calibrated. If not, the forecaster has concrete evidence that their judgement needs adjustment.
Without predefined resolution rules, those comparisons become unreliable because the target itself keeps moving. A forecast that cannot be resolved consistently cannot improve future judgement.
Resolution rules therefore serve a simple but powerful purpose: they transform uncertain beliefs into claims that reality can fairly test. That makes them one of the most practical habits for anyone trying to think more clearly about an uncertain future.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: metaculus.com
Link: https://www.metaculus.com/question-writing/Source snippet
Question writing and submission guidelinesSpecifying precise resolution criteria. Resolution criteria are the backbone of any forecasting...
-
Source: metaculus.com
Link: https://www.metaculus.com/help/question-checklist/Source snippet
Metaculus Question Approval ChecklistWhen naming a resolution source, are we trying to forecast that particular source more than the "tru...
-
Source: metaculus.com
Link: https://www.metaculus.com/Source snippet
Metaculus is an online forecasting platform and aggregation engine working to improve human reasoning and coordination on topics...
-
Source: metaculus.com
Link: https://www.metaculus.com/faq/Source snippet
Metaculus FAQFrequently asked questions about Metaculus, including basics, question types, resolution processes, predictions, scoring, an...
-
Source: metaculus.com
Link: https://www.metaculus.com/help/guidelines/Source snippet
munity GuidelinesAccepting community-suggested questions that are ready to go live for forecasting. Mediating discussions about question...
-
Source: metaculus.com
Link: https://www.metaculus.com/notebooks/22486/a-primer-on-the-metaculus-scoring-rule/Source snippet
A Primer on the Metaculus Scoring RuleFeb 27, 2021 — Our first criterion is that our Scoring Rule be “proper”, which means that in expect...
-
Source: metaculus.com
Link: https://www.metaculus.com/notebooks/7095/add-your-own-alt-protein-questions/Source snippet
Add your own Alt-Protein questionsWrite clear and precise resolution conditions. Resolution conditions are the backbone of any forecastin...
-
Source: metaculus.com
Link: https://www.metaculus.com/how-to-forecast/Source snippet
How to forecast on MetaculusTo predict, share the probability you give the outcome as a number between 0.1% and 99.9%. On the question pa...
-
Source: metaculus.com
Title: how metaculus leverages crowd forecasting
Link: https://www.metaculus.com/notebooks/40619/how-metaculus-leverages-crowd-forecasting/Source snippet
Nov 7, 2025 — Break the decision into resolvable sub-questions with crisp resolution criteria and timelines. Draft and pre-mortem. Write...
-
Source: metaculus.com
Title: question writing checklist obsolete
Link: https://www.metaculus.com/notebooks/3418/question-writing-checklist-obsolete/Source snippet
Question writing checklist (obsolete)5 Jun 2023 — Please suggest below items you would like to see on a checklist automatically displayed...
-
Source: metaculus.com
Title: forecasting our world in data the next 100 years
Link: https://www.metaculus.com/notebooks/14965/forecasting-our-world-in-data-the-next-100-years/Source snippet
Forecasting Our World in Data: The Next 100 Years1 Feb 2023 — A key component of a well-written forecasting question is its resolution cr...
-
Source: metaculus.com
Title: self resolving question methods
Link: https://www.metaculus.com/notebooks/4031/self-resolving-question-methods/Source snippet
Self-resolving question methodsJun 5, 2023 — And the resolution criteria would say: On Jan 1, 2000, 3 people will be randomly chosen from...
-
Source: rethinkpriorities.org
Title: types of specification problems in forecasting
Link: https://rethinkpriorities.org/research-area/types-of-specification-problems-in-forecasting/Source snippet
Rethink PrioritiesTypes of specification problems in forecasting20 Jul 2021 — Metaculus has a guide on how to write unambiguous and usefu...
Additional References
-
Source: goodjudgment.com
Link: https://goodjudgment.com/Source snippet
Good Judgment: See the future sooner with SuperforecastingWith a subscription to FutureFirst you have access to dozens of daily updated f...
-
Source: ijcb.org
Link: https://www.ijcb.org/journal/v3n4/inflation-forecast-based-rules-and-indeterminancy-puzzle-and-resolutionSource snippet
Inflation-Forecast-Based Rules and IndeterminancyOur resolution is that central banks should be viewed as following "Calvo-type" inflatio...
-
Source: proceedings.iclr.cc
Link: https://proceedings.iclr.cc/paper_files/paper/2025/file/5107a33432b9a5cffafd0a53ef6c6a18-Paper-Conference.pdfSource snippet
CHECKS FOR LANGUAGE MODEL...– Retain the same resolution date as the source forecasting question. User Prompt: • The source forecasting...
-
Source: ecb.europa.eu
Link: https://www.ecb.europa.eu/pub/pdf/scpwps/ecbwp643.pdfSource snippet
We propose viewing central banks as following 'Calvo-type' inflation-forecast-based interest rate rules...Read more...
-
Source: papers.neurips.cc
Link: https://papers.neurips.cc/paper_files/paper/2025/file/2277c7c3b112fcc6031a6f0d832df2a0-Paper-Datasets_and_Benchmarks_Track.pdfSource snippet
However, existing forecast- ing benchmarks lack comprehensive confidence [assessment]({{ 'assessment/' | relative_url }}), focusing on limited...
-
Source: forum.effectivealtruism.org
Title: A higher score indicates greater relative accuracy. A score of 0
Link: https://forum.effectivealtruism.org/posts/F2stjK9wHSy3HPEC9/q2-ai-benchmark-results-pros-maintain-clear-leadSource snippet
effectivealtruism.orgQ2 AI Benchmark Results: Pros Maintain Clear LeadOct 28, 2025 — The score is zero-sum (i.e., the sum of all particip...
-
Source: youtube.com
Title: Exploring the intricacies of prediction markets | David Chee | EAG London 23
Link: https://www.youtube.com/watch?v=S4Oa_CD5a0wSource snippet
Exploring the intricacies of prediction markets explores the specific challenges of designing rigorous resolution criteria to make foreca...
-
Source: aiimpacts.org
Link: https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/Source snippet
ce in the same broad domain · Making more predictions on the same question...Read more...
-
Source: lesswrong.com
Link: https://www.lesswrong.com/posts/oJ6wXoBqxJjHhPLLu/an-examination-of-metaculus-resolved-ai-predictions-andSource snippet
An examination of Metaculus' resolved AI predictions and...Jul 20, 2021 — Of all 41 date questions, 4 were predicted to be 25%-50% to re...
-
Source: lesswrong.com
Title: resolutions to the challenge of resolving forecasts
Link: https://www.lesswrong.com/posts/JnDEAmNhSpBRpjD8L/resolutions-to-the-challenge-of-resolving-forecastsSource snippet
11 Mar 2021 — We can be inflexible with resolution criteria, and always specify exactly what number or fact will be used for the resoluti...
Topic Tree