Within Deliberate Practice
Why prediction logs make thinking measurable
Prediction logs make reasoning practice measurable by recording forecasts, evidence, updates and results in one place.
On this page
- What to record before the outcome is known
- How scoring reveals recurring judgement errors
- Using updates as practice rather than embarrassment
Page outline Jump by section
Introduction
Prediction logs turn reasoning into something that can be measured instead of merely remembered. Rather than asking whether a decision “felt sensible”, they require you to state a forecast before the outcome is known, explain why you believe it, record your confidence as a probability, and later compare that judgement with reality. This creates a permanent record of how well your thinking matches the world rather than your memory of it.
Within deliberate practice for reasoning, prediction logs are valuable because they generate repeated, scoreable judgements. They expose patterns that are otherwise difficult to notice, such as chronic overconfidence, excessive caution, reluctance to update beliefs, or relying on weak evidence. Research from forecasting tournaments, including the Good Judgment Project, has shown that keeping score and repeatedly refining probabilistic judgements can improve forecasting performance over time.[Good Judgment+2AI Impacts]goodjudgment.comGood JudgmentThe Science Of SuperforecastingProven methods. Powerful results. Good Judgment research discovered four keys to accurate for…
What to record before the outcome is known
The usefulness of a prediction log depends far more on what is written before an outcome is known than on commentary afterwards. Once people know the result, hindsight bias makes previous beliefs seem more obvious than they really were.
A practical prediction entry usually includes:
- The prediction itself. Phrase it so that it will clearly be true or false. Ambiguous predictions cannot be scored consistently.
- A probability estimate. Instead of “probably”, assign a number such as 30%, 65% or 90%. Probability estimates reveal calibration—whether events assigned a 70% chance really happen about seven times in ten.
- A deadline. Specify exactly when the prediction will be judged.
- The evidence available now. Separate observations from interpretations. Record the facts you relied upon rather than reconstructing them later.
- Alternative explanations. Briefly note what would make the opposite outcome plausible. This discourages tunnel vision.
- Conditions for revision. State what new evidence would cause you to increase or decrease your confidence before the deadline.
For example, instead of writing, “I think sales will improve”, a prediction log might record:
“There is a 65% probability that quarterly sales will exceed last year’s equivalent quarter. Confidence is based on current order volume and announced product launches. If two major customers cancel before month-end, confidence falls below 50%.”
The important feature is that every part of the reasoning is timestamped before reality provides the answer.
How scoring reveals recurring judgement errors
Prediction logs become useful only when forecasts are scored systematically rather than remembered selectively.
Forecasting research commonly uses the Brier score, a proper scoring rule that compares predicted probabilities with actual outcomes. Lower scores indicate better probabilistic forecasting because they reward both accuracy and well-calibrated confidence. Proper scoring rules encourage honest probability estimates rather than exaggerated certainty designed to appear impressive.[Wikipedia+2scattered-thoughts.net]WikipediaBrier scoreBrier score
Over dozens or hundreds of predictions, scores begin to reveal stable reasoning habits rather than isolated mistakes. Typical patterns include:
- Overconfidence. Events predicted with 90% confidence occur far less frequently than expected.
- Underconfidence. Correct forecasts are consistently assigned probabilities that are too low.
- Poor discrimination. Almost every forecast receives similar probabilities, regardless of evidence quality.
- Failure to distinguish evidence strength. Weak and strong cases receive nearly identical confidence levels.
Calibration is especially informative. Suppose every forecast assigned an 80% probability is later examined. If only about half actually occur, confidence is inflated. If nearly all occur, confidence may be too conservative. Good calibration means long-run frequencies match stated probabilities, not that every individual prediction succeeds.[PMC]pmc.ncbi.nlm.nih.gov# 4: A BrierOn misconceptions about the Brier score in binary prediction…by L Hoessly · 2026 · Cited by 18 — Calibration refers to how well pre…
It is also important not to overinterpret a single score. Researchers emphasise that the Brier score combines several aspects of forecast quality and is influenced by the difficulty of the prediction task. It should therefore be used to compare repeated forecasts in similar settings rather than treated as a universal measure of intelligence or reasoning ability.[PubMed]pubmed.ncbi.nlm.nih.govOn misconceptions about the Brier score in binary…by L Hoessly · 2026 · Cited by 19 — The Brier score was shown to reflect not o…
Using updates as practice rather than embarrassment
Many people resist recording prediction updates because changing one’s mind can feel like admitting failure. Forecasting research suggests the opposite.
Good forecasters revise beliefs whenever meaningful new evidence appears. The goal is not consistency with previous opinions but increasing correspondence with reality. Forecasts are therefore living judgements rather than declarations of loyalty to an initial view.[Good Judgment]goodjudgment.comGood JudgmentThe Science Of SuperforecastingProven methods. Powerful results. Good Judgment research discovered four keys to accurate for…
A useful update records three things:
- what new evidence appeared;
- how much confidence changed;
- why the evidence justified that change.
For instance:
DateProbabilityReason2 June45%Initial evidence mixed10 June62%New economic data stronger than expected18 June54%Competitor announcement weakens original case
Reviewing these updates later often exposes recurring habits. Some people hardly update until the answer is obvious. Others swing dramatically after every headline. Both patterns become visible only when changes are logged over time.
The objective is not to minimise revisions but to make them evidence-based. A prediction log rewards thoughtful updating because every revision itself becomes observable practice in reasoning under uncertainty.
What prediction logs teach that memory cannot
Human memory edits past beliefs surprisingly well. After outcomes become known, people frequently remember having been “almost certain” of whatever happened, even when earlier confidence was much lower. This hindsight bias makes informal self-assessment unreliable.
Prediction logs bypass this problem because they preserve:
- original confidence levels;
- original reasoning;
- the evidence available at the time;
- every subsequent revision.
Instead of debating whether a decision was “reasonable”, the learner can inspect exactly what information was available and whether confidence was proportional to that evidence.
Over time, this archive becomes a personalised dataset of thinking errors. Unlike general advice about cognitive biases, it identifies the biases that repeatedly affect one individual’s judgements.
Building a useful prediction log
The most effective logs favour many small predictions over a few dramatic ones. Frequent forecasts provide enough observations to detect patterns and shorten the feedback cycle.
Suitable practice questions include:
- Will this project finish by its stated deadline?
- Will a published study replicate?
- Will a policy proposal pass within six months?
- Will an investment thesis outperform a benchmark over a specified period?
- Will a software feature be completed by the next release?
The questions matter less than their clarity. Good prediction questions have an observable outcome, a defined resolution date and a probability estimate recorded in advance.
Consistency also matters more than complexity. A simple spreadsheet or notebook that captures forecasts, evidence, updates and outcomes is usually sufficient. The cumulative record—not any single prediction—is what gradually improves judgement.
As forecasting researchers often note, the discipline of keeping score distinguishes genuine learning from the comforting illusion of improvement. Without recorded predictions and objective feedback, confidence can increase indefinitely while accuracy remains unchanged.[Good Judgment+2Good Judgment]goodjudgment.comgood judgments 2025 in reviewGood JudgmentGood Judgment's 2025 in ReviewJanuary 6, 2026 — Superforecasters always keep score. As we turn to 2026 at Good Judgment Inc…
Amazon book picks
Further Reading
Books and field guides related to Why prediction logs make thinking measurable. Use these as the next step if you want deeper reading beyond the article.
Superforecasting
Directly explains forecasting, calibration, prediction tracking, and improving judgement through feedback.
Thinking, Fast and Slow
Provides the cognitive bias foundation that prediction logs help reveal and correct.
The Scout Mindset
Focuses on updating beliefs, intellectual honesty, and better calibrated reasoning.
How to Measure Anything
Encourages measurable thinking and probabilistic estimation instead of intuition alone.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
1.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC7333631/
Source snippet
reality check: Evidence from a small pool of...by I Katsagounos · 2020 · Cited by 24 — The evidence in support of the superforecasting h...
2.
Source: Wikipedia
Title: Brier score
Link:https://en.wikipedia.org/wiki/Brier_score
3.
Source: scattered-thoughts.net
Title: notes on superforecasting the art and science of prediction
Link:https://www.scattered-thoughts.net/blog/2016/01/28/notes-on-superforecasting-the-art-and-science-of-prediction
Source snippet
This is a scoring rule with the useful property that the maximum expected score is given...Read more...
4.
Source: pmc.ncbi.nlm.nih.gov
Title: # 4: A Brier
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC12818272/
Source snippet
On misconceptions about the Brier score in binary prediction...by L Hoessly · 2026 · Cited by 18 — Calibration refers to how well pre...
5.
Source: Wikipedia
Title: The [Good Judgment]({{ ‘good-judgment/’ | relative_url }}) Project
Link:https://en.wikipedia.org/wiki/The_Good_Judgment_Project
Source snippet
The Good Judgment ProjectThe Good Judgment Project (GJP) is an organization dedicated to harnessing the wisdom of the crowd to forecas...
6.
Source: goodjudgment.com
Link:https://goodjudgment.com/about/the-science-of-superforecasting/
Source snippet
Good JudgmentThe Science Of SuperforecastingProven methods. Powerful results. Good Judgment research discovered four keys to accurate for...
7.
Source: aiimpacts.org
Link:https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/
Source snippet
In the tournament, six teams assigned probabilistic answers to...Read more...
8.
Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/41567946/
Source snippet
On misconceptions about the Brier score in binary...by L Hoessly · 2026 · Cited by 19 — The Brier score was shown to reflect not o...
9.
Source: goodjudgment.com
Title: good judgments 2025 in review
Link:https://goodjudgment.com/good-judgments-2025-in-review/
Source snippet
Good JudgmentGood Judgment's 2025 in ReviewJanuary 6, 2026 — Superforecasters always keep score. As we turn to 2026 at Good Judgment Inc...
Published: January 6, 2026
10.
Source: aiimpacts.org
Link:https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project-an-accompanying-blog-post/
Source snippet
Evidence on good forecasting practices from the...2 Jul 2019 — Tetlock used something very much like a Brier score in this tournament, b...
11.
Source: gjopen.com
Link:https://www.gjopen.com/
Source snippet
Good Judgment® OpenA forecasting services firm that equips corporate, government, and non-governmental decision-makers with the benefit o...
12.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10788705/
Source snippet
aware training to improve deep learning model...by T Dawood · 2023 · Cited by 51 — Our final calibration measure is the Brier Score (BS)...
13.
Source: youtube.com
Link:https://www.youtube.com/watch?v=pedNak4S9IE
Source snippet
s developed. Like Daniel Kahneman's...
14.
Source: good-judgment.thinkific.com
Link:https://good-judgment.thinkific.com/pages/faq
Source snippet
Fundamentals FAQThe course is designed to develop six fundamental forecasting skills. Our research shows that each of these skills contri...
Additional References
15.
Source: translationalethics.com
Link:https://www.translationalethics.com/faq/
Source snippet
Frequently Asked QuestionsA Brier score is a measure of forecasting skill.... Another is by Phil Tetlock and Daniel Gardner (Superforeca...
16.
Source: scikit-learn.org
Link:https://scikit-learn.org/stable/modules/calibration.html
17.
Source: merriam-webster.com
Link:https://www.merriam-webster.com/dictionary/confidence
18.
Source: medium.com
Link:https://medium.com/data-science/model-calibration-explained-a-visual-guide-with-code-examples-for-beginners-55f368bafe72
19.
Source: edge.org
Title: philip tetlock edge master class 2015 a short course in superforecasting class i
Link:https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-i
Source snippet
A Short Course in Superforecasting, Class IAug 17, 2015 — You could give even greater weight to being accurate further off than the Brier...
20.
Source: stats.stackexchange.com
Title: is it true that log loss requires probabilities to be calibrated or otherwise i
Link:https://stats.stackexchange.com/questions/675995/is-it-true-that-log-loss-requires-probabilities-to-be-calibrated-or-otherwise-i
Source snippet
This question needs to be more focused. It is not currently accepting answers. Want to improve this question? Guide the asker to update...
21.
Source: cameronrwolfe.substack.com
Title: confidence calibration for deep networks why and how e2cd4fe4a086
Link:https://cameronrwolfe.substack.com/p/confidence-calibration-for-deep-networks-why-and-how-e2cd4fe4a086
Source snippet
Calibration for Deep Networks: Why and How?Confidence calibration is defined as the ability of some model to provide an accurate probabil...
22.
Source: researchgate.net
Title: (PDF) Superforecasting: The Art and Science of Prediction
Link:https://www.researchgate.net/publication/304924623_Superforecasting_The_Art_and_Science_of_Prediction_By_Philip_Tetlock_and_Dan_Gardner
Source snippet
Jul 5, 2016 — This is an excellent book to read. It is not only informative, as it should be for a book on forecasting, but it is highly...
23.
Source: researchgate.net
Link:https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions
Source snippet
May 25, 2015 — Mean standardized [Brier scores]({{ 'brier-scores/' | relative_url }}) for superforecasters (Supers) and the two comparison groups in Years 2 and 3 for forecasts...
Published: May 25, 2015
24.
Source: openreview.net
Link:https://openreview.net/forum?id=34xYxTTiM0
Source snippet
The Cross-Entropy (CE) loss is widely used for calibrator training, which enforces...
Topic Tree