Within Calibration
Can Your Forecast Actually Be Scored?
A good prediction journal starts with questions that are specific enough to resolve and narrow enough to teach you something.
On this page
- What makes a prediction resolvable
- Turning vague claims into clear questions
- Common framing mistakes that ruin feedback
Page outline Jump by section
Introduction
A prediction journal only teaches you anything if your forecasts can later be judged against reality. That means every forecast should answer a question with a clear resolution, a defined deadline, and an outcome that independent observers would agree on. Vague predictions such as “the economy will struggle” or “this project will probably succeed” may feel insightful, but they generate little useful feedback because there is no objective way to determine whether they were correct. Research from forecasting tournaments led by Philip Tetlock and colleagues consistently relies on carefully specified, resolvable questions because accurate scoring is the foundation of calibration and improvement.[Wharton Faculty Platform]faculty.wharton.upenn.edu2015 superforecastersWharton Faculty Platform2015—superforecasters.pdf - Wharton Faculty PlatformMay 18, 2015 — by B Mellers · 2015 · Cited by 332 — Brier s…
Writing scorable questions is therefore not administrative detail. It is the mechanism that turns intuition into measurable performance. The clearer the question, the easier it becomes to compare your confidence with reality, identify patterns of overconfidence or underconfidence, and improve future judgement.
What makes a prediction resolvable?
A forecast is resolvable when a future observer can answer it without debating what the predictor “really meant”. Good forecasting platforms and research projects impose strict resolution criteria for exactly this reason.[Commoncog]commoncog.comHow Do You Evaluate Your Own Predictions?How Do You Evaluate Your Own Predictions?December 16, 2019 — 17 Dec 2019 — This post provides a comprehensive summary of the tec…
A well-written forecast normally contains five elements:
- A single outcome. Ask one question at a time rather than combining several claims.
- A precise deadline. State exactly when the forecast stops being updated and when it will be judged.
- An observable criterion. The answer should depend on publicly observable facts or clearly defined evidence.
- Binary resolution whenever possible. “Yes” or “No” questions are much easier to score than subjective assessments.
- A stable resolution source. Decide in advance which source will determine the outcome if there is disagreement.
For example:
Poor question
Will artificial intelligence transform education?
Problems:
- No deadline.
- “Transform” is undefined.
- Multiple interpretations are possible.
Scorable version
Will at least 30% of secondary schools in England report using generative AI tools in regular classroom instruction by 31 December 2028, according to the annual Department for Education survey?
Even if the prediction proves difficult, everyone knows how it will be resolved.
Turning vague claims into clear questions
Many forecasting errors begin before probabilities are assigned. The question itself is too broad to generate meaningful feedback.
A useful way to rewrite forecasts is to identify four components:
Vague statementBetter forecasting questionInflation will stay high.Will annual UK CPI inflation exceed 4% in the Office for National Statistics release for December 2026?The company will do well.Will Company X report positive year-on-year revenue growth in its FY2027 annual report?This product launch will succeed.Will Product X sell more than one million units worldwide within six months of release?The negotiations will fail.Will the parties publicly announce that negotiations have ended without a signed agreement before 1 October 2026?
Notice that each revision introduces measurable thresholds, a deadline, and an identifiable source.
This approach also forces you to think about what evidence would genuinely change your mind. If you cannot describe what success or failure looks like, you are probably forecasting an impression rather than an event.
Choose questions that teach you something
Not every scorable question is equally valuable.
Forecasting research suggests concentrating on questions that are uncertain but answerable rather than either obvious or effectively unknowable. Tetlock describes this as working in the “Goldilocks zone”: difficult enough that judgement matters, but not so uncertain that evidence cannot discriminate between better and worse forecasts.[Good Judgment]goodjudgment.comGood JudgmentTen Commandments for Aspiring SuperforecastersThese commandments describe behaviors that have been “experimentally demonstra…
Questions are especially useful when they:
- involve decisions you regularly make;
- repeat often enough to build a track record;
- depend on evidence you can observe before resolution;
- encourage updating rather than one-off guessing.
For example, someone managing software projects might repeatedly forecast delivery dates, customer renewals, or hiring outcomes. After dozens of such forecasts, patterns become visible that would never emerge from isolated predictions.
Define the resolution rule before making the forecast
One of the easiest ways to corrupt a prediction journal is to change the scoring rules after the outcome is known.
Write the resolution criteria before assigning a probability.
A simple template is:
- Question: Will Event X occur?
- Resolution date: 30 September 2027.
- Resolution source: Organisation Y’s official report.
- Resolves Yes if: Conditions A and B are both satisfied.
- Resolves No if: The deadline passes without those conditions.
- Resolves Void if: The specified report is permanently cancelled.
Precommitting to these rules prevents hindsight bias and motivated reinterpretation. Forecasting tournaments use similar procedures because disagreements about wording can otherwise overwhelm meaningful evaluation.[Commoncog]commoncog.comHow Do You Evaluate Your Own Predictions?How Do You Evaluate Your Own Predictions?December 16, 2019 — 17 Dec 2019 — This post provides a comprehensive summary of the tec…
Common framing mistakes that ruin feedback
Several recurring mistakes make forecasts difficult or impossible to score.
Bundling multiple predictions together
Questions like:
Will inflation fall, unemployment remain low, and interest rates be cut?
contain three separate forecasts. One may be correct while the others fail.
Instead, create three independent questions.
Using subjective language
Words such as:
- successful
- major
- significant
- dramatic
- soon
- reliable
usually require interpretation.
Replace them with measurable quantities, dates, or explicit thresholds.
Leaving the deadline open
Predictions without end dates eventually become impossible to falsify.
“Bitcoin will exceed $200,000.”
Compared with:
“Will Bitcoin close above US$200,000 on any trading day before 31 December 2028?”
The second version can actually be resolved.
Allowing moving goalposts
Avoid writing questions that permit changing definitions after new information appears.
For example:
The project counts as successful if management seems happy.
Instead, specify measurable criteria before the project finishes.
Predicting trends rather than events
Forecasts like:
Public opinion will shift.
are difficult to score.
Convert them into measurable outcomes:
Will support for Policy X exceed 55% in the next nationally representative YouGov poll conducted before 1 June 2027?
Build questions that work with probabilistic scoring
Scorable questions also make statistical evaluation possible.
Measures such as the Brier score compare the probability you assigned with the actual outcome. These scoring rules reward honest probability estimates rather than exaggerated confidence. However, they only work when each forecast has a clearly defined outcome that eventually resolves as true or false.[Wikipedia+2Corporate Research Forum]WikipediaBrier scoreBrier score
For example:
- 80% probability that an event occurs.
- Event happens.
- Forecast receives a relatively good score.
If the same event fails to occur, the high-confidence prediction receives a much worse score, providing useful feedback about calibration rather than simply whether the prediction was right.
Without unambiguous questions, that feedback loop breaks down.
A practical checklist before recording a forecast
Before adding a new entry to your prediction journal, ask:
- Can an independent person decide whether this is correct?
- Is there a specific resolution date?
- Have I defined exactly what counts as “yes” and “no”?
- Is there one primary outcome rather than several bundled together?
- Have I identified the source that will resolve the question?
- Would I still agree with these rules after learning the outcome?
If every answer is “yes”, the question is ready for a probability estimate and, later, objective scoring. That is what allows a prediction journal to become a genuine learning tool rather than a collection of memorable guesses.
Amazon book picks
Further Reading
Books and field guides related to Can Your Forecast Actually Be Scored?. Use these as the next step if you want deeper reading beyond the article.
The Signal and the Noise
Illustrates how good forecasts depend on clear evidence and measurable outcomes.
How to Measure Anything
Supports writing measurable, objectively resolvable questions.
The Scout Mindset
Reinforces honest evaluation and learning from forecasting feedback.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
1.
Source: commoncog.com
Title: How Do You Evaluate Your Own Predictions?
Link:https://commoncog.com/how-do-you-evaluate-your-own-predictions/
Source snippet
How Do You Evaluate Your Own Predictions?December 16, 2019 — 17 Dec 2019 — This post provides a comprehensive summary of the tec...
Published: December 16, 2019
2.
Source: Wikipedia
Title: Brier score
Link:https://en.wikipedia.org/wiki/Brier_score
3.
Source: Wikipedia
Title: The [Good Judgment]({{ ‘good-judgment/’ | relative_url }}) Project
Link:https://en.wikipedia.org/wiki/The_Good_Judgment_Project
Source snippet
The Good Judgment ProjectThe Good Judgment Project (GJP) is an organization dedicated to harnessing the wisdom of the crowd to forecas...
4.
Source: faculty.wharton.upenn.edu
Title: 2015 superforecasters
Link:https://faculty.wharton.upenn.edu/wp-content/uploads/2015/07/2015—superforecasters.pdf
Source snippet
Wharton Faculty Platform2015---superforecasters.pdf - Wharton Faculty PlatformMay 18, 2015 — by B Mellers · 2015 · Cited by 332 — Brier s...
Published: May 18, 2015
5.
Source: goodjudgment.com
Link:https://goodjudgment.com/philip-tetlocks-10-commandments-of-superforecasting/
Source snippet
Good JudgmentTen Commandments for Aspiring SuperforecastersThese commandments describe behaviors that have been “experimentally demonstra...
6.
Source: crforum.co.uk
Link:https://www.crforum.co.uk/wp-content/uploads/2016/10/2016_Schoemaker_Tetlock_Superforecasting-HBR-May-2016.pdf
Source snippet
rporate Research ForumSuperforecasting: How to Upgrade Your Company's...by PJH Schoemaker · 2016 · Cited by 74 — The closer to zero th...
Additional References
7.
Source: thedecisionlab.com
Link:https://thedecisionlab.com/thinkers/political-science/philip-tetlock
Source snippet
Philip TetlockTetlock is a psychology professor and researcher who is fascinated by decision-making processes and the attributes required...
8.
Source: edge.org
Link:https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii
Source snippet
A Short Course in Superforecasting, Class II24 Aug 2015 — A short course in superforecasting, Class II. Tournaments: Prying Open Closed M...
9.
Source: aiimpacts.org
Link:https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/
Source snippet
ce in the same broad domain · Making more predictions on the same question...Read more...
10.
Source: scattered-thoughts.net
Title: notes on superforecasting the art and science of prediction
Link:https://www.scattered-thoughts.net/blog/2016/01/28/notes-on-superforecasting-the-art-and-science-of-prediction
Source snippet
Tetlock uses the Brier score to score predictions. This is a scoring rule with the useful property...Read more...
11.
Source: youtube.com
Title: How to Enter | What a Winning Forecast Actually Looks Like
Link:https://www.youtube.com/watch?v=LcjHd4Febpg
Source snippet
Introducing a New Forecast Question Type: Conditional Pairs...
12.
Source: youtube.com
Title: Introducing a New Forecast Question Type: Conditional Pairs
Link:https://www.youtube.com/watch?v=ZvZeA0qvZXg
Source snippet
Improve Your Decision-Making with Your Own Brier Score...
13.
Source: youtube.com
Title: Improve Your Decision-Making with Your Own Brier Score
Link:https://www.youtube.com/watch?v=sL39bKyHcLI
Source snippet
Forecasting Without Magic: Metrics, Scenarios, Bayes...
14.
Source: youtube.com
Title: Forecasting Without Magic: Metrics, Scenarios, Bayes
Link:https://www.youtube.com/watch?v=tmKp9c1fWVg
Source snippet
Superforecasting Summary | Philip E. Tetlock...
15.
Source: youtube.com
Link:https://www.youtube.com/watch?v=k9KLo_slozY
Topic Tree