Within Feedback
Why Vague Predictions Do Not Teach Much
Putting a number on a prediction turns fuzzy confidence into something that can be checked, scored and improved.
On this page
- Why probabilities beat vague language
- How to define a forecast question
- Checking calibration without overreacting
Page outline Jump by section
Introduction
Many judgement errors persist because people never state clearly what they expected. Saying that an outcome is “likely”, “possible” or “unlikely” feels informative, but those words are flexible enough to be reinterpreted after the event. Assigning a probability instead—such as a 30% chance, a 70% chance or a 95% chance—turns a vague opinion into a forecast that can later be checked. That simple shift creates the feedback loop needed to improve judgement: expectations become measurable, confidence becomes testable, and repeated forecasts reveal whether you are systematically overconfident, underconfident or well calibrated. Research from forecasting tournaments and decision science shows that keeping score in this way is one of the most effective ways to improve predictive judgement over time.[AI Impacts]aiimpacts.orgAI ImpactsEvidence on good forecasting practices from the…Tetlock says that reading the news and generating probabilities isn't enough…
Why probabilities beat vague language
Words such as “probably”, “good chance” or “almost certain” sound precise but mean different things to different people. One person’s “likely” may imply a 60% chance, while another hears 90%. This ambiguity makes learning difficult because nobody can later agree on what was actually predicted.
A numerical probability removes that ambiguity. If you predict that there is a 70% chance a project will finish on time, the forecast can eventually be judged against reality. One forecast proves little, but a collection of forecasts allows patterns to emerge. If events assigned 70% probability occur roughly seven times out of ten over many similar predictions, your confidence is well calibrated. If they occur only half the time, you have been overconfident. If they occur nine times out of ten, you have been too cautious.[Society for Judgment and Decision Making]sjdm.orgSociety for Judgment and Decision MakingA model-based approach for the analysis of the calibration…The calibration of probability or c…
This is why forecasting researchers distinguish between expressing an opinion and making a forecast. Opinions may persuade other people, but forecasts create evidence about the quality of your judgement.
How to define a forecast question
Not every statement can be assigned a useful probability. Good forecast questions share three features.
- A clear event. Everyone should agree whether it happened. “The product launch will be successful” is too vague. “The product launches before 30 September” is clear.
- A defined time horizon. Every forecast needs a deadline for resolution.
- A binary outcome. Although reality is often complex, scoring becomes much easier when the question resolves as either yes or no.
For example:
- “There is a 40% chance our supplier misses the August delivery deadline.”
- “There is a 75% chance inflation will remain above 3% at year end.”
- “There is a 20% chance this job applicant accepts the offer within two weeks.”
Each statement specifies what counts as success or failure before the outcome is known. That prevents hindsight from quietly changing the meaning afterwards.
Professional forecasting tournaments, including those behind the Good Judgment Project, rely on carefully written binary questions for exactly this reason. Precise questions make meaningful scoring possible and reduce disputes over whether a prediction was actually correct.[Wikipedia]WikipediaThe Good Judgment ProjectThe Good Judgment Project
Checking calibration without overreacting
The goal is not to make every prediction correct. A forecast with a 30% probability should be wrong most of the time. Good forecasting means matching confidence to reality.
Look for long-run patterns
Calibration only becomes meaningful across many forecasts. If you predict twenty events at 80% probability, you would expect around sixteen to occur. A run of three unexpected failures does not automatically mean your judgement is poor.
Looking at groups of forecasts avoids overreacting to chance. Random variation affects individual predictions, but repeated forecasts reveal systematic tendencies.
Separate calibration from discrimination
Good judgement has two related but different qualities.
- Calibration asks whether your stated probabilities match observed frequencies.
- Discrimination asks whether you successfully distinguish likely events from unlikely ones.
Someone who predicts 50% for every question may end up reasonably calibrated overall if about half the events occur, but the forecasts are not useful because they fail to separate strong possibilities from weak ones. Good forecasters are both calibrated and willing to move away from 50% when the evidence supports doing so.[Coefficient Giving]coefficientgiving.orgefforts to improve the accuracy of our judgments and forecastsThe improvement in mean Brier scores from probability…Read more…
Avoid chasing every surprise
People naturally remember spectacular misses and forget routine successes. This creates a temptation to radically change forecasting habits after a single unexpected outcome.
A better approach is to review forecasts periodically rather than emotionally. If repeated reviews show that predictions around 90% succeed only 70% of the time, confidence should become more cautious. If predictions around 30% almost never happen, confidence may have become too pessimistic.
The objective is gradual improvement, not constant adjustment after every surprise.
Keeping score with proper scoring rules
Simply counting correct and incorrect predictions ignores confidence. A forecast of 51% and one of 99% both receive the same outcome if the event happens, even though the second represented far greater confidence.
Forecasting therefore uses proper scoring rules, which reward honest probability estimates rather than exaggerated certainty. The most widely used is the Brier score, which measures the squared difference between the predicted probability and the eventual outcome. Lower scores indicate better forecasting performance. Importantly, the Brier score encourages reporting your genuine belief rather than inflating confidence to appear decisive.[Wharton Faculty Platform]faculty.wharton.upenn.edu2015 superforecastersWharton Faculty Platform2015—superforecasters.pdf - Wharton Faculty Platformby B Mellers · 2015 · Cited by 319 — Brier scores are the a…
For example:
ForecastOutcomeInterpretation90% and the event happensStrong, accurate prediction90% and the event does not happenSevere overconfidence55% and the event happensMild confidence, modest reward50% every timeSafe from extreme errors, but rarely informative
Because scoring reflects both correctness and confidence, it reveals whether mistakes come from poor information or from misplaced certainty.
What probability forecasting changes in everyday thinking
Assigning probabilities changes reasoning before the outcome arrives, not merely afterwards.
Instead of asking, “Do I think this will happen?”, you begin asking:
- What evidence would justify moving from 60% to 75%?
- What information would make me lower this estimate?
- What base rate should anchor my initial judgement?
- What would surprise me enough to revise my view?
These questions encourage continuous updating instead of defending an initial opinion. Studies of high-performing forecasters consistently find that they revise probabilities incrementally as new evidence appears rather than waiting until they feel completely certain. Training, regular feedback and repeated scoring all contribute to better calibration over time.[Good Judgment+2Stanford University]goodjudgment.comGood JudgmentThe Science Of SuperforecastingProven methods. Powerful results. Good Judgment research discovered four keys to accurate for…
Common mistakes
Several habits reduce the value of probability forecasts:
- Using only extreme probabilities. Reserving 0% and 100% for events that are logically impossible or certain avoids pretending to know more than you do.
- Failing to define the event. Vague forecasts cannot be scored fairly.
- Judging yourself after only a few forecasts. Calibration requires repeated observations.
- Changing the prediction after the fact. Forecasts should be recorded before outcomes are known.
- Treating probabilities as guarantees. A 70% forecast still implies that the event should fail about three times in ten.
The discipline is therefore less about predicting perfectly than about making expectations explicit enough that experience can genuinely improve judgement. Over time, numerical forecasts expose recurring patterns in confidence that vague language tends to conceal, making thinking progressively more accurate rather than merely more certain.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
1.
Source: web.stanford.edu
Link:https://web.stanford.edu/~knutson/jdm/mellers15.pdf
Source snippet
Stanford UniversityIdentifying and Cultivating Superforecasters as a Method of...by B Mellers · 2015 · Cited by 315 — Brier scores are t...
2.
Source: Wikipedia
Title: The Good Judgment Project
Link:https://en.wikipedia.org/wiki/The_Good_Judgment_Project
3.
Source: Wikipedia
Link:https://en.wikipedia.org/wiki/Calibration
Source snippet
CalibrationCalibration is the comparison of measurement values delivered by a device under test with those of a calibration standard o...
4.
Source: Wikipedia
Title: Brier score
Link:https://en.wikipedia.org/wiki/Brier_score
Source snippet
Brier scoreThe Brier score is a strictly proper scoring rule that measures the accuracy of probabilistic predictions.Read more...
5.
Source: aiimpacts.org
Link:https://aiimpacts.org/evidence-on-good-forecasting-practices-from-the-good-judgment-project/
Source snippet
AI ImpactsEvidence on good forecasting practices from the...Tetlock says that reading the news and generating probabilities isn't enough...
6.
Source: sjdm.org
Link:https://sjdm.org/~baron/journal/11/m01/m01.html
Source snippet
Society for Judgment and Decision MakingA model-based approach for the analysis of the calibration...The calibration of probability or c...
7.
Source: faculty.wharton.upenn.edu
Title: 2015 superforecasters
Link:https://faculty.wharton.upenn.edu/wp-content/uploads/2015/07/2015—superforecasters.pdf
Source snippet
Wharton Faculty Platform2015---superforecasters.pdf - Wharton Faculty Platformby B Mellers · 2015 · Cited by 319 — Brier scores are the a...
8.
Source: coefficientgiving.org
Title: efforts to improve the accuracy of our judgments and forecasts
Link:https://coefficientgiving.org/research/efforts-to-improve-the-accuracy-of-our-judgments-and-forecasts/
Source snippet
The improvement in mean Brier scores from probability...Read more...
9.
Source: goodjudgment.com
Link:https://goodjudgment.com/about/the-science-of-superforecasting/
Source snippet
Good JudgmentThe Science Of SuperforecastingProven methods. Powerful results. Good Judgment research discovered four keys to accurate for...
10.
Source: bol.com
Link:https://www.bol.com/nl/nl/f/superforecasting/9200000045386346/
Source snippet
, Philip Tetlock | 9781847947154 | Boeken | bolBoek geeft inzichten in hoe we goede en minder goede voorspellingen doen, maar vooral wat...
11.
Source: goodjudgment.com
Link:https://goodjudgment.com/
Source snippet
Good Judgment: See the future sooner with SuperforecastingTetlock and Mellers co-founded Good Judgment Inc to provide forecasting service...
12.
Source: goodjudgment.com
Link:https://goodjudgment.com/services/online-training/
Source snippet
Fundamentals-Focused Online Training For ForecastersThis concise, self-paced online course targets the fundamental skills needed to start...
13.
Source: goodjudgment.com
Link:https://goodjudgment.com/open-minded-forecasting-in-a-deeply-polarized-world/
Source snippet
Open-Minded Forecasting in a Deeply Polarized WorldStudies based on the Good Judgment Project (GJP) found that being an “actively open-mi...
14.
Source: goodjudgment.com
Link:https://goodjudgment.com/wp-content/uploads/2022/10/Superforecaster-Accuracy.pdf
Source snippet
calibration curve shows whether we can take Good Judgment's forecasts at face value. With 323 questions scored and analyzed, the Superfor...
15.
Source: youtube.com
Link:https://www.youtube.com/watch?v=pedNak4S9IE
Source snippet
Superforecasting | Philip TetlockThe pundits we all listen to are no better at predictions than a “dart-throwing chimp,” and they are rou...
16.
Source: marketing.wharton.upenn.edu
Link:https://marketing.wharton.upenn.edu/wp-content/uploads/2015/04/WALTERS-DANIELS-JMP.pdf
Source snippet
Unknowns: A Critical Determinant of Confidence...by DJ Walters · Cited by 92 — [overconfidence]({{ 'overconfidence/' | relative_url }}) has been so influential is because it has...
Additional References
17.
Source: goodreads.com
Link:https://goodreads.com/book/show/23995360-superforecasting
Source snippet
Superforecasting: The Art and Science of PredictionSuperforecasting offers the first demonstrably effective way to improve our ability to...
18.
Source: corporate.jasoncollins.blog
Link:https://corporate.jasoncollins.blog/better-forecasting
Source snippet
jasoncollins.blog25 Better forecastingIn this page, I examine techniques to improve forecasting accuracy, primarily through evidence from...
19.
Source: stefanverstraeten.medium.com
Link:https://stefanverstraeten.medium.com/secrets-of-superforecasting-why-the-world-and-your-business-need-better-predictions-d7fa0615322b
Source snippet
of Superforecasting — Why the world and your...If finance leaders want to build better prediction capability, there's several powerful s...
20.
Source: andrewclark.co.uk
Link:https://andrewclark.co.uk/all-media/superforecasting
Source snippet
SuperForecastingExpress your judgment as precisely as you can, using a finely grained scale of probability. Update to reflect the latest...
21.
Source: pmc.ncbi.nlm.nih.gov
Title: The superforecasting hypothesis is challenged under real-life scarcity
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC7333631/
Source snippet
reality check: Evidence from a small pool of...by I Katsagounos · 2020 · Cited by 23 — The study contributes to the stream of literature...
22.
Source: edge.org
Link:https://www.edge.org/conversation/philip_tetlock-edge-master-class-2015-a-short-course-in-superforecasting-class-ii
Source snippet
A Short Course in Superforecasting, Class IIAug 24, 2015 — The interesting thing is the exercises themselves only require about an hour a...
23.
Source: scattered-thoughts.net
Title: notes on superforecasting the art and science of prediction
Link:https://www.scattered-thoughts.net/blog/2016/01/28/notes-on-superforecasting-the-art-and-science-of-prediction/
Source snippet
Notes on 'Superforecasting: The Art and Science of...Jan 28, 2016 — A weather forecaster who makes predictions using the seasonal averag...
24.
Source: researchgate.net
Link:https://www.researchgate.net/figure/Mean-standardized-Brier-scores-for-superforecasters-Supers-and-the-two-comparison_fig1_277087515
Source snippet
hich reality is coded as 1 for the event and 0 otherwise), ranging from 0 (...Read more...
25.
Source: forum.effectivealtruism.org
Title: evidence on good forecasting practices from the good 1
Link:https://forum.effectivealtruism.org/posts/W94KjunX3hXAtZvXJ/evidence-on-good-forecasting-practices-from-the-good-1
Source snippet
on good forecasting practices from the...15 Feb 2019 — Tetlock says that reading the news and generating probabilities isn't enough; you...
26.
Source: crforum.co.uk
Title: 2016 Schoemaker Tetlock Superforecasting HBR May 2016
Link:https://www.crforum.co.uk/wp-content/uploads/2016/10/2016_Schoemaker_Tetlock_Superforecasting-HBR-May-2016.pdf
Source snippet
Superforecasting: How to Upgrade Your Company's...by PJH Schoemaker · 2016 · Cited by 71 — To improve prediction capability, companies s...
Published: May 2016
Topic Tree