Within Calibration
The Predictions Worth Writing Down
The most useful journal entries are concrete, repeated predictions about decisions where better calibration would change behavior.
On this page
- Why repeated small forecasts beat rare dramatic ones
- Work, research, planning, and external event examples
- How to avoid obvious or unscorable questions
Page outline Jump by section
Introduction
A prediction journal improves judgement only if it captures the kinds of forecasts that can change future decisions. The most valuable entries are not dramatic guesses about elections, pandemics or technological breakthroughs. They are repeated, concrete forecasts about situations you encounter often enough to learn from: project deadlines, hiring decisions, research hypotheses, customer behaviour, negotiations or personal planning. Repetition creates feedback, and feedback allows calibration—bringing your confidence into better alignment with reality. Research on forecasting consistently shows that improvement comes from making clear probabilistic predictions, reviewing outcomes and updating beliefs, not from occasionally being right about spectacular events.[Good Judgment+2Wiley Online Library]goodjudgment.comGood JudgmentThe Science Of SuperforecastingGood Judgment research discovered four keys to accurate forecasting: talent-spotting, trainin…
The Predictions Worth Writing Down
A useful prediction journal asks one question repeatedly: “If I become better at forecasting this type of event, will I make better decisions?”
Many possible predictions fail this test. Guessing tomorrow’s lottery numbers, celebrity news or isolated political surprises may be entertaining, but they rarely improve the choices you make. By contrast, forecasting recurring decisions lets you discover systematic patterns in your thinking.
Good prediction topics usually have four characteristics:
- They occur regularly enough to generate many observations.
- The outcome can be resolved objectively.
- The prediction influences an actual decision.
- Similar situations arise again, allowing lessons to transfer.
Forecasting researchers emphasise that skill develops through repeated practice with feedback rather than isolated one-off judgements. Forecasting tournaments intentionally present participants with many questions because calibration improves through accumulation rather than memorable anecdotes.[Good Judgment+2PMC]goodjudgment.comGood JudgmentThe Science Of SuperforecastingGood Judgment research discovered four keys to accurate forecasting: talent-spotting, trainin…
Why Repeated Small Forecasts Beat Rare Dramatic Ones
Large, unusual events create poor learning environments. They happen infrequently, involve many uncontrollable variables and often cannot be compared with previous experiences. Even if your prediction turns out correct, you may not know whether it reflected genuine insight or luck.
Small recurring forecasts provide much richer feedback.
Instead of asking:
- “Will artificial general intelligence arrive before 2035?”
consider asking:
- “Will this machine learning experiment improve validation accuracy by at least 2%?”
- “Will the reviewer request major revisions?”
- “Will this feature reduce support tickets within one month?”
Instead of:
- “Will the economy enter recession next year?”
consider:
- “Will our sales pipeline convert at least 18% of qualified leads this quarter?”
- “Will procurement costs exceed our budget by more than 5%?”
After dozens of similar predictions, patterns emerge. You may discover that you are consistently overconfident about deadlines, overly pessimistic about customer adoption or too optimistic when evaluating your own ideas. Those discoveries directly improve future planning.
Philip Tetlock’s work distinguishes forecasting skill from isolated prediction success. Reliable judgement comes from repeated probabilistic estimation, continual updating and disciplined evaluation rather than occasional impressive forecasts.[Wharton Faculty Platform+2econtalk.org]faculty.wharton.upenn.eduWharton Faculty Platform ExpertIn the long run (…Read more…
Focus on Decisions You Actually Control
Prediction journals become more valuable when forecasts support governance of your own work rather than passive observation.
Useful categories include:
Project management
- Will this milestone finish before Friday?
- Will integration testing uncover more than five critical bugs?
- Will the supplier deliver within the agreed window?
These forecasts improve scheduling, contingency planning and resource allocation.
[Research]researchgate.netPDF) Superforecasting: The Art and Science of Prediction5 Jul 2016 — This is an excellent book to read. It is not only informative, as it should be for a book on forecasting, but it is highly e…
Researchers can forecast whether an experiment will replicate, whether data collection will finish on time or whether a hypothesis will survive analysis. Recording confidence before seeing results reduces hindsight bias and encourages honest evaluation.
Professional judgement
Managers can estimate:
- whether a candidate will accept an offer,
- whether a proposal will be approved,
- whether a client will renew,
- whether estimated effort will exceed actual effort.
Because these decisions recur frequently, forecasting exposes persistent biases.
Personal planning
Examples include:
- finishing a report before a deadline,
- maintaining an exercise routine for four weeks,
- completing reading goals,
- keeping travel within budget.
Although these predictions concern personal behaviour, they remain measurable and directly affect future planning.
Include External Events That Affect Decisions
Not every useful forecast concerns your own actions. External events deserve attention when they influence meaningful decisions.
Examples include:
- Will inflation exceed a threshold that changes investment strategy?
- Will a regulator publish new guidance before contract negotiations?
- Will a competitor release a product before your planned launch?
- Will interest rates remain unchanged before refinancing?
These questions connect outside uncertainty to concrete choices. Their purpose is not to demonstrate world knowledge but to improve decision timing, contingency planning and resource allocation.
Forecasting organisations often focus on clearly defined external events because they produce objective feedback while encouraging careful probability estimates instead of binary certainty.[Good Judgment]goodjudgment.comGood JudgmentThe Science Of SuperforecastingGood Judgment research discovered four keys to accurate forecasting: talent-spotting, trainin…
How to Avoid Questions That Teach You Nothing
Many prediction journals gradually become collections of interesting trivia. Several warning signs indicate that a question is unlikely to improve judgement.
Avoid questions that are:[gjopen.com]gjopen.comt by 100 so that your probabilities range between 0 (0…Read more…
- Too vague. “Will this project succeed?” leaves too much room for reinterpretation.
- Impossible to resolve. “Will people eventually appreciate this idea?” has no obvious endpoint.
- Dependent on hidden definitions. Everyone should know exactly what counts as success or failure.
- Extremely rare. Learning from a prediction that resolves once every decade is painfully slow.
- Pure entertainment. If the answer would not change future behaviour, it adds little value.
A good forecasting question specifies:[goodjudgment.com]goodjudgment.comGood JudgmentThe Science Of SuperforecastingGood Judgment research discovered four keys to accurate forecasting: talent-spotting, trainin…
- the event,
- the deadline,
- the resolution rule,
- the probability,
- and the information available when the prediction was made.
Clear definitions make later evaluation fair and prevent subconscious reinterpretation after the outcome is known. Forecasting practice consistently stresses explicit time horizons and operational definitions because ambiguity weakens learning.[Summrize]summrize.comSuperforecasting by Philip Tetlock BookSuperforecasting by Philip Tetlock Book SummarySpecify clear time horizons and definitions for all forecast questions; Track pred…
Build Families of Similar Forecasts
Individual predictions matter less than collections of related ones.
Instead of recording one forecast about software delivery, build a category:
- release dates,
- bug counts,
- estimation accuracy,
- customer adoption,
- post-release defects.
Likewise, a researcher might maintain separate groups for:
- experiment duration,
- statistical significance,
- reviewer feedback,
- recruitment rates.
Grouping similar forecasts allows meaningful comparison across time. You begin asking questions such as:
- Am I always optimistic about schedules?
- Do I underestimate uncertainty early in projects?
- Which kinds of evidence reliably change my mind?
These recurring patterns are far more informative than isolated successes or failures.
Prefer Questions With Clear Feedback Loops
Fast feedback accelerates improvement.
Questions resolving within days or weeks allow frequent calibration. Questions resolving over several years may still matter strategically but contribute less to everyday learning because opportunities for correction arrive slowly.
A balanced prediction journal often contains:
- many short-term operational forecasts,
- some medium-term planning forecasts,[emilkirkegaard.dk]emilkirkegaard.dkPhilip E. Tetlock Expert Political Judgment How Bookos.orgExpert Political JudgmentSome long-term forecasts that experts offered will not come due until 2026. But most of the data are tabulated…
- only a few long-term strategic forecasts.
This balance keeps the journal active while still capturing important longer-range decisions.
Studies of forecasting competitions likewise rely on repeated forecasts with defined resolution periods, allowing forecasters to update beliefs and learn systematically over time rather than waiting years between evaluations.[PMC]pmc.ncbi.nlm.nih.govSuperforecasting reality check: Evidence from a small pool of…by I Katsagounos · 2020 · Cited by 23 — The questions provided had a…
Judge Success by Better Decisions, Not More Correct Predictions
The ultimate purpose of choosing forecast questions is not to maximise the percentage of correct guesses. Well-calibrated forecasters sometimes assign 40% probabilities to events that occur and 80% probabilities to events that fail. What matters is whether confidence appropriately reflects uncertainty.
A useful prediction journal therefore asks:
- Did this forecast improve my decision?
- Did repeated forecasts reveal a recurring bias?
- Did I become better calibrated over time?
If the answer is yes, the journal is fulfilling its purpose.
Forecast evaluation methods such as the Brier score reward both accuracy and well-calibrated probabilities, encouraging forecasters to distinguish genuine uncertainty from unwarranted confidence. Training studies show that regular feedback improves this calibration over repeated forecasting cycles.[gjopen.com+2Wiley Online Library]gjopen.comt by 100 so that your probabilities range between 0 (0…Read more…
Amazon book picks
Further Reading
Books and field guides related to The Predictions Worth Writing Down. Use these as the next step if you want deeper reading beyond the article.
Thinking in Bets
Shows how to make and review decisions under uncertainty using probabilistic thinking.
The Signal and the Noise
Explains why repeated, measurable forecasts improve predictive skill.
The Art of Thinking Clearly
Introduces common thinking errors that prediction journals can expose.
Algorithms to Live By
Encourages structured decision-making and repeatable learning across everyday choices.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
1.
Source: onlinelibrary.wiley.com
Title: Online Library Automated calibration training for forecasters
Link:https://onlinelibrary.wiley.com/doi/full/10.1002/bdm.2334
Source snippet
Wiley Online LibraryAutomated calibration training for forecasters - Stone - 2023by ER Stone · 2023 · Cited by 4 — In two studies, we inv...
2.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC7333631/
Source snippet
Superforecasting reality check: Evidence from a small pool of...by I Katsagounos · 2020 · Cited by 23 — The questions provided had a...
3.
Source: econtalk.org
Link:https://www.econtalk.org/philip-tetlock-on-superforecasting/
Source snippet
Philip Tetlock on Superforecasting - Econlib19 Dec 2015 — In this conversation, Tetlock discusses the meaning, reliability, and usefulnes...
4.
Source: gjopen.com
Link:https://www.gjopen.com/faq
Source snippet
t by 100 so that your probabilities range between 0 (0...Read more...
5.
Source: summrize.com
Title: Superforecasting by Philip Tetlock Book
Link:https://www.summrize.com/books/super-forecasting-summary
Source snippet
Superforecasting by Philip Tetlock Book SummarySpecify clear time horizons and definitions for all forecast questions; Track pred...
6.
Source: harry-cheslaw.medium.com
Title: super forecasting dd146e441c1c
Link:https://harry-cheslaw.medium.com/super-forecasting-dd146e441c1c
Source snippet
medium.comSuper-Forecasting. By Philip Tetlock and Dan GardnerSuper-Forecasting uses this previous work as a foundation to further our ab...
7.
Source: goodjudgment.com
Link:https://goodjudgment.com/about/the-science-of-superforecasting/
Source snippet
Good JudgmentThe Science Of SuperforecastingGood Judgment research discovered four keys to accurate forecasting: talent-spotting, trainin...
8.
Source: faculty.wharton.upenn.edu
Title: Wharton Faculty Platform Expert
Link:https://faculty.wharton.upenn.edu/wp-content/uploads/2012/04/Tetlock_2005-EPJ-chapter-1.pdf
Source snippet
In the long run (...Read more...
9.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10189590/
Source snippet
improves forecasting - PMC - NIHby DN Ferreiro · 2023 · Cited by 4 — We hypothesize that compromise forecasts, defined as the average pre...
10.
Source: forum.effectivealtruism.org
Link:https://forum.effectivealtruism.org/posts/pnpnqA4hijnr59p7d/efforts-to-improve-the-accuracy-of-our-judgments-and
Source snippet
[Good Judgment]({{ 'good-judgment/' | relative_url }}) Project forecasting tournaments: Training yielded significant...
11.
Source: blas.com
Link:https://blas.com/superforecasting/
Source snippet
by Dan Gardner and Philip Tetlock17 Nov 2015 — Great read on how to become a much better forecaster by making a forecast, measuring it, r...
12.
Source: faculty.wharton.upenn.edu
Title: SSRN id3779404
Link:https://faculty.wharton.upenn.edu/wp-content/uploads/2022/03/SSRN-id3779404.pdf
Source snippet
Brier score on questions where everyone else is close to zero is far less impressive than close-to-zero score on questions where everyone...
Additional References
13.
Source: bedfordconsulting.com
Link:https://bedfordconsulting.com/eight-forecasting-best-practice-tips/
Source snippet
Eight forecasting best practice tips | BlogRead our blog to find out eight best practise tips and better understanding where your busines...
14.
Source: statmodeling.stat.columbia.edu
Link:https://statmodeling.stat.columbia.edu/2024/11/01/calibration-is-sometimes-sufficient-for-trusting-predictions-what-does-this-tell-us-when-human-experts-use-model-predictions/
Source snippet
1 Nov 2024 — “Calibrated predictions have the property that it is optimal for a decision maker to optimize assuming that the prediction i...
15.
Source: empslocal.ex.ac.uk
Link:https://empslocal.ex.ac.uk/people/staff/dbs202/publications/2019/siegertstephenson.pdf
Source snippet
Recalibration and Multimodel Combinationby S Siegert · Cited by 13 — By using statistical models to correct forecast errors of dynamical...
16.
Source: 4castplus.com
Title: top 5 planning tips for accurate project cash flow forecasting part 2
Link:https://4castplus.com/top-5-planning-tips-for-accurate-project-cash-flow-forecasting-part-2/
Source snippet
Top 5 Planning Tips for Accurate Project Cash Flow...6 Feb 2026 — Accurate project cash flow forecasting during execution using discipli...
17.
Source: osf.io
Link:https://osf.io/download/n5czv
Source snippet
le scoring rules, we use data from the Good Judgment Project that were collected during the...Read more...
18.
Source: researchgate.net
Title: (PDF) Superforecasting: The Art and Science of Prediction
Link:https://www.researchgate.net/publication/304924623_Superforecasting_The_Art_and_Science_of_Prediction_By_Philip_Tetlock_and_Dan_Gardner
Source snippet
5 Jul 2016 — This is an excellent book to read. It is not only informative, as it should be for a book on forecasting, but it is highly e...
19.
Source: emilkirkegaard.dk
Title: Philip E. Tetlock Expert Political Judgment How Bookos.org
Link:https://emilkirkegaard.dk/en/wp-content/uploads/Philip_E.Tetlock_Expert_Political_Judgment_HowBookos.org.pdf
Source snippet
Expert Political JudgmentSome long-term forecasts that experts offered will not come due until 2026. But most of the data are tabulated...
20.
Source: lifeitself.org
Title: Superforecasting, Tetlock and Gardner (Notes)Chapter 3
Link:https://lifeitself.org/blog/notes-on-tetlock-and-gardners-superforecasting
Source snippet
In the mid-1980s Tetlock began a research programme to learn what sets the best forecasters apart. He recruited experts...Read more...
21.
Source: lesswrong.com
Title: question tracking accuracy of personal forecasts
Link:https://www.lesswrong.com/posts/R22HQJBiMnaSrr6cN/question-tracking-accuracy-of-personal-forecasts
Source snippet
[Question] Tracking accuracy of personal forecasts20 Mar 2019 — Brier score would be great at telling me how accurate I am, but not what...
22.
Source: sobrief.com
Link:https://sobrief.com/books/expert-political-judgment
Source snippet
Expert Political Judgment | Summary, Audio, Quotes, FAQ10 Aug 2025 — Expert Political Judgment explores the accuracy of political experts...
Topic Tree