Within Predictions
Stop explaining results after the fact
A simple prediction log can stop post-hoc storytelling after metrics, experiments, or project feedback arrive.
On this page
- What to write before opening results
- How prior forecasts expose weak assumptions
- How teams can review surprises without blame
Page outline Jump by section
Introduction
Prediction logs are a simple way to prevent one of the biggest obstacles to analytical improvement: rewriting your own thinking after the outcome is already known. Before opening a dashboard, reading experiment results, checking project metrics or reviewing stakeholder feedback, you record what you expect to happen, how confident you are, and why. When the actual results arrive, you compare them with the written prediction rather than with your memory.
This habit matters because people are naturally vulnerable to hindsight bias—the tendency to believe that outcomes were more obvious and predictable after they occur than they really were. A prediction log creates an objective record of your pre-result reasoning, making it much easier to identify which assumptions were genuinely useful and which merely seemed convincing in retrospect. Research on prediction, learning and metacognition suggests that committing to expectations before feedback changes how people process surprises and improves calibration over time. [PMC+2Carnegie Mellon University]pmc.ncbi.nlm.nih.govPMCPredicting as a learning strategyNIHby G Brod · 2021 · Cited by 77 — Initial evidence suggests that predicting boosts surprise about unexpected answers, which leads…
What to write before opening results
A prediction log should be brief enough that it becomes routine but detailed enough to expose your reasoning. The goal is not to forecast perfectly but to leave a clear trail of how you reached your expectation.
For each important result, record:
- The prediction: What do you expect to happen?
- Confidence: Express uncertainty as a percentage or probability range rather than “certain” or “unlikely”.
- Reasoning: Which evidence, pattern or assumption led you to that expectation?
- Key uncertainties: What might prove you wrong?
- Decision implication: If your prediction is wrong, what would that suggest about your current model?
For example, before reviewing an online experiment:
ItemExample entryPredictionVariant B will increase conversion by about 4%.Confidence65%ReasonPrevious mobile friction was reduced.Main uncertaintyReturning users may behave differently.If wrongNavigation changes may matter less than pricing visibility.
Notice that the prediction includes both an expected direction and an explanation. Recording reasons is usually more valuable than recording guesses alone because it allows later inspection of the underlying mental model rather than simply counting wins and losses.
How prior forecasts expose weak assumptions
Without written predictions, feedback often creates an illusion that the outcome was obvious all along. Once the numbers appear, people unconsciously edit their memory of what they believed beforehand. Hindsight bias makes learning from experience much harder because there is no reliable baseline against which to compare reality. Carnegie Mellon University+2Carlson School of Management [cmu.edu]cmu.eduOutcome FeedbackCarnegie Mellon UniversityOutcome Feedback: Hindsight and Informationby SJ Hoch · 1989 · Cited by 247 — Although "hindsight bias" researc…
Prediction logs interrupt that process in several useful ways.
They separate luck from reasoning. A correct forecast reached for poor reasons is different from a correct forecast based on an accurate understanding of the system. Likewise, an incorrect prediction supported by sensible reasoning may reveal random variation rather than flawed thinking.
They reveal recurring blind spots. After several weeks or months, patterns often emerge:
- consistently overestimating positive outcomes
- excessive confidence despite mixed accuracy
- relying too heavily on recent examples
- repeatedly ignoring operational constraints
- underestimating uncertainty in complex projects
These patterns are difficult to detect from memory because memory itself becomes distorted by outcomes.
They improve calibration. Calibration refers to how closely confidence matches actual accuracy. Someone whose 70% confidence predictions are correct roughly seven times out of ten is well calibrated. Forecasting research consistently identifies calibration as an important component of good judgement, and keeping explicit confidence records provides the raw material needed to evaluate it. [Effective Altruism Forum]forum.effectivealtruism.orgEffective Altruism ForumEfforts to Improve the Accuracy of Our Judgments and…October 25, 2016 — 25 Oct 2016 — An important component o…
Turning analytics into better thinking instead of better storytelling
Analytics dashboards generate large amounts of feedback, but feedback alone does not guarantee learning. Without prior expectations, every unexpected metric invites an explanation constructed after the fact.
Prediction logs reverse the order.
Instead of asking:
Why did this happen?
the first question becomes:
Compared with what I expected, what specifically surprised me?
That small change has important consequences.
Suppose a marketing campaign produces lower engagement than expected. Rather than immediately constructing a narrative, the analyst compares today’s data with the written forecast:
- Which prediction was accurate? [innerdrive.co.uk]innerdrive.co.uke correct answer, they have a better memory of the correct response.Read more…
- Which assumption failed?
- Was confidence justified?
- Which evidence received too much weight?
- What evidence was ignored?
This comparison turns every result into a test of the analyst’s reasoning rather than merely an explanation of the outcome.
Research on prediction as a learning strategy suggests that committing to an expectation before receiving feedback increases attention to discrepancies between expectation and reality. Those discrepancies become informative because they highlight where the mental model requires revision. [PMC]pmc.ncbi.nlm.nih.govPMCPredicting as a learning strategyNIHby G Brod · 2021 · Cited by 77 — Initial evidence suggests that predicting boosts surprise about unexpected answers, which leads…
How teams can review surprises without blame
Prediction logs are particularly valuable in organisations because they shift review discussions away from personal judgement and towards reasoning quality.
Instead of asking who was right, teams can examine:
- Which assumptions held?
- Which assumptions failed?
- Which uncertainties were recognised beforehand?
- Which signals were overlooked?
- Which future indicators deserve greater attention?
This encourages learning rather than defensive behaviour.
A practical meeting structure looks like this:
- Read the original predictions before viewing explanations.
- Compare forecasts with actual outcomes.
- Identify the largest surprises.
- Discuss why expectations differed from reality.
- Record one or two concrete updates to future forecasting practice.
This sequence resembles structured after-action reviews, where organisations first establish intended outcomes, compare them with what actually occurred, examine why differences emerged, and identify improvements for future work. The emphasis is on organisational learning rather than assigning fault. [Better Evaluation]betterevaluation.orgBetter EvaluationAfter action reviewThe after action review (AAR) is a simple method for facilitating an assessment of organisational per…
Common mistakes that reduce the value of prediction logs
Several habits undermine the usefulness of prediction logs even when people remember to create them.
Predictions that are too vague. Statements such as “performance should improve” are difficult to evaluate. A forecast should be specific enough that reasonable observers would agree whether it succeeded.
No confidence estimate. Without recording confidence, analysts cannot distinguish justified uncertainty from unwarranted certainty.
No explanation. A prediction without reasons provides little insight into the thinking process.
Editing predictions afterwards. Any modification after seeing results defeats the purpose of preserving an unbiased record.
Only reviewing failures. Correct predictions also deserve inspection. Success sometimes reflects sound reasoning, but it can also result from chance. Analysing both correct and incorrect forecasts prevents overconfidence.
Building the habit into everyday work
Prediction logs work best when they become part of ordinary workflows rather than occasional exercises.
Useful opportunities include:
- before opening weekly performance dashboards
- before reading customer satisfaction surveys
- before reviewing A/B test outcomes
- before project retrospectives
- before receiving stakeholder feedback
- before examining financial or operational metrics
Many organisations find that a simple shared template is sufficient. The important feature is consistency rather than sophistication. A spreadsheet, notebook or shared document can all work, provided entries are timestamped before results become available.
Over time, the accumulated record becomes more valuable than any individual prediction. Instead of relying on memory, teams gain evidence about how their judgement evolves, which assumptions repeatedly fail, and where confidence is consistently too high or too low. That long-term feedback loop is what transforms prediction logs from a record-keeping exercise into a practical tool for improving analytical thinking.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCPredicting as a learning strategy
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8642250/Source snippet
NIHby G Brod · 2021 · Cited by 77 — Initial evidence suggests that predicting boosts [surprise]({{ 'surprise/' | relative_url }}) about unexpected answers, which leads...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCResponse-based outcome predictions and confidence
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8121545/Source snippet
by R Frömer · 2021 · Cited by 68 — We propose that people leverage insights from response-based performance monitoring – outcome predi...
-
Source: cmu.edu
Title: Outcome Feedback
Link: https://www.cmu.edu/dietrich/sds/docs/loewenstein/OutcomeFeedback.pdfSource snippet
Carnegie Mellon UniversityOutcome Feedback: Hindsight and Informationby SJ Hoch · 1989 · Cited by 247 — Although "hindsight bias" researc...
-
Source: carlsonschool.umn.edu
Title: vohs et al 2012 hindsight bias
Link: https://carlsonschool.umn.edu/sites/carlsonschool.umn.edu/files/2026-01/vohs-et-al-2012-hindsight-bias.pdfSource snippet
Carlson School of ManagementScience Perspectives on Psychologicalby NJ Roese · 2012 · Cited by 926 — Hindsight bias occurs when people fe...
-
Source: forum.effectivealtruism.org
Link: https://forum.effectivealtruism.org/posts/pnpnqA4hijnr59p7d/efforts-to-improve-the-accuracy-of-our-judgments-andSource snippet
Effective Altruism ForumEfforts to Improve the Accuracy of Our Judgments and...October 25, 2016 — 25 Oct 2016 — An important component o...
Published: October 25, 2016
-
Source: betterevaluation.org
Link: https://www.betterevaluation.org/methods-approaches/methods/after-action-reviewSource snippet
Better EvaluationAfter action reviewThe after action review (AAR) is a simple method for facilitating an [assessment]({{ 'assessment/' | relative_url }}) of organisational per...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8763848/Source snippet
Impact of Cognitive Biases on Professionals' Decision...by V Berthet · 2022 · Cited by 305 — First, the literature reviewed shows that a...
-
Source: innerdrive.co.uk
Link: https://www.innerdrive.co.uk/blog/predicting-or-retrieving/Source snippet
e correct answer, they have a better memory of the correct response.Read more...
-
Source: link.springer.com
Link: https://link.springer.com/article/10.3758/s13421-020-01012-wSource snippet
hindsight bias | Memory & Cognitionby R Ackerman · 2020 · Cited by 29 — Our results show that learning the correct answers distorts memor...
Additional References
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/357667792_Improving_Judgments_of_Existential_Risk_Better_Forecasts_Questions_Explanations_PoliciesSource snippet
(PDF) Improving Judgments of Existential Risk21 Jan 2022 — PDF | Forecasting [tournaments]({{ 'tournaments/' | relative_url }}) are misaligned with the goal of producing action...
-
Source: my.chartered.college
Link: https://my.chartered.college/research-hub/research-informed-practice-after-action-reviews/Source snippet
chartered.collegeResearch-informed practice: After-action reviewsAn after-action review (AAR) is a group process designed to give you cle...
-
Source: thedecisionlab.com
Link: https://thedecisionlab.com/biases/hindsight-biasSource snippet
Hindsight BiasThe hindsight bias describes our tendency to look back at an unpredictable event and think it was easily predictable.Read more...
-
Source: jbsfm.org
Link: https://jbsfm.org/vol3no1/biases-in-managerial-decision-making–overconfidence–status-quo–anchoring–hindsight–availability/Source snippet
Biases in Managerial Decision Making: Overconfidence...30 Jul 2021 — Hindsight is a bias. Hindsight effect is when people claim that, af...
-
Source: aiimpacts.org
Link: https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-[good-judgmentSource snippet
Evidence on good forecasting practices from the...2 Jul 2019 — Experience and data from the Good Judgment Project (GJP) provide importan...
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=IIrkQluCAsQSource snippet
Prediction journal decision journal hindsight bias Decision Journaling: Learn From Every Choice The Mastery Project...
-
Source: researchgate.net
Title: 350390831 Predicting as a learning strategy
Link: https://www.researchgate.net/publication/350390831_Predicting_as_a_learning_strategySource snippet
(PDF) Predicting as a learning strategy29 Mar 2021 — This article attempts to delineate the procedural and mechanistic characteristics of...
-
Source: chrisquigley.co.uk
Link: https://www.chrisquigley.co.uk/blog/a-metacognitive-strategy-of-predicting-teaching-tentative-language-to-overcome-the-fear-of-being-wrong/Source snippet
A Metacognitive Strategy of Predicting: Teaching Tentative...16 May 2024 — Emphasise the learning that comes from examining why a predic...
Published: May 2024
-
Source: microsoft.com
Title: Prediction strategies without loss
Link: https://www.microsoft.com/en-us/research/publication/prediction-strategies-without-loss/Source snippet
Microsoft Research1 Jan 2012 — We obtain essentially zero loss with respect to the special expert and optimal loss/regret tradeoff, impro...
-
Source: fasterthannormal.co
Link: https://fasterthannormal.co/mental-models/hindsight-biasSource snippet
ires accurate feedback: I predicted X, Y happened...
Topic Tree