Within Feedback
What Forecasting Tournaments Teach Daily Decisions
Forecasting tournament practices show how training, teamwork and scoring can improve ordinary judgement loops.
On this page
- Why explicit predictions beat general impressions
- Which tournament habits transfer to everyday judgement
- Where formal scoring becomes too heavy
Page outline Jump by section
Introduction
The Good Judgment Project showed that everyday judgement can be improved by turning hunches into testable forecasts. Its lesson is simple but demanding: state what you expect, attach a probability, write down why, update when evidence changes, and later score how well your confidence matched reality. In forecasting tournaments run under the US intelligence community’s Aggregative Contingent Estimation programme, Good Judgment’s methods combined training, collaboration, performance tracking and statistical aggregation to improve geopolitical prediction accuracy. The most useful everyday takeaway is not to copy a full tournament at home, but to borrow its feedback loop: make your expectations explicit enough that reality can teach you. [IARPA+2WashU Research Profiles]iarpa.govACEThe goal of the ACE Program is to dramatically enhance the accuracy, precision, and timeliness of intelligence forecasts for a br…
Why explicit predictions beat general impressions
Most ordinary decisions hide a forecast. Taking a new job contains a forecast about fit, pay, learning and stress. Planning a project contains a forecast about time, cost and likely obstacles. Deciding whether to raise a concern in a meeting contains a forecast about how others will respond. The problem is that these forecasts are often left in vague language: “probably fine”, “unlikely”, “soon”, “risky”, “worth it”. Vague language protects the ego because it is hard to mark later as right or wrong.
Forecasting tournaments forced a sharper format. Participants made probabilistic forecasts on clearly defined questions with future resolution dates, and their answers were scored. Good Judgment Open explains the Brier score as a measure of how far a probability forecast was from reality: lower is better, and a 70% forecast on a yes-or-no question is judged differently from a 95% forecast. That distinction matters because it rewards calibrated confidence, not merely being directionally right. [Good Judgment Open]gjopen.comGood Judgment OpenFrequently Asked Questions (FAQ)Roughly, the Brier score indicates how far away from the truth your forecast was. The B…
This is the first transferable habit: replace general impressions with forecastable statements. “This project is risky” becomes “I think there is a 35% chance we miss the deadline by more than two weeks.” “The client will like it” becomes “I would put a 70% probability on approval without a major redesign.” The wording is less elegant, but the feedback is far better. When the outcome arrives, you can ask whether your confidence was sensible rather than merely whether fortune favoured you.
What made the Good Judgment Project work
The Good Judgment Project was not just a crowd guessing exercise. Research on its performance points to several mutually reinforcing drivers: short training, team discussion, tracking of strong performers, repeated practice and aggregation of forecasts. A 2014 Psychological Science paper found support for training, teaming and tracking as drivers of accuracy, with probability training encouraging reference-class thinking, bias correction and averaging multiple estimates. [WashU Research Profiles]profiles.wustl.eduWashU Research ProfilesPsychological Strategies for Winning a Geopolitical…by B Mellers · 2014 · Cited by 434 — Forecasting is often v…
A later paper on cultivating superforecasters described high performers as partly discovered and partly created. The strongest forecasters tended to combine cognitive ability and open-minded style with task-specific skill, motivation, frequent updating and an environment that rewarded accuracy rather than status. That is an important historical lesson: the tournament did not merely identify people who were already good; it created conditions under which good judgement could compound. [Stanford University]web.stanford.eduOpen source on stanford.edu.
Training also mattered even when it was brief. A four-year analysis of geopolitical forecasting tournaments found that a cognitive-debiasing training module lasting less than an hour improved Brier-score accuracy by 6% to 11% compared with a control condition. That is modest enough to be believable and large enough to matter: judgement improved not through personality transformation, but through repeated use of better habits. [Cambridge University Press & Assessment]cambridge.orgOpen source on cambridge.org.
Which tournament habits transfer to everyday judgement
The strongest everyday lesson is to build small feedback loops around decisions that repeat. The Good Judgment Project dealt with geopolitical questions, but the habits transfer best to ordinary domains where outcomes arrive often enough to review: hiring, project planning, sales estimates, study plans, personal finance choices, health routines, meeting preparation and relationship expectations.
Useful habits include:
- Use numbers when the decision matters. A rough probability is better than a polished adjective. “60%” forces more discipline than “likely”.
- Write the reason before the result is known. This protects the original judgement from hindsight bias and makes later review useful.
- Use the outside view. Ask what usually happens in similar cases before focusing on the special details of this case.
- Update visibly. A changed forecast is not a failure; it is evidence that new information has been absorbed.
- Compare with others. Good Judgment’s team settings helped forecasters share information and challenge rationales. Everyday versions include asking colleagues for independent estimates before discussion.
- Track a small set of recurring predictions. Ten well-reviewed forecasts teach more than dozens of casual opinions.
The most practical version is a decision journal with probabilities. For example, before committing to a three-month project, record: “75% chance we finish the core version by 30 September; 40% chance we need extra design support; 20% chance the main blocker is stakeholder delay.” At the review date, the point is not to punish error. It is to notice patterns: overconfidence, neglected bottlenecks, excessive caution, or forecasts that improve when you consult a particular colleague.
Why teamwork helps without becoming groupthink
One surprising lesson from Good Judgment is that teams can improve judgement when they are structured around evidence rather than status. In the tournament research, team collaboration allowed forecasters to share information and discuss the reasoning behind their beliefs; elite teams of top performers improved calibration and resolution. [WashU Research Profiles]profiles.wustl.eduWashU Research ProfilesPsychological Strategies for Winning a Geopolitical…by B Mellers · 2014 · Cited by 434 — Forecasting is often v…
For everyday decisions, the crucial move is to separate independent judgement from group discussion. Ask people for their initial probabilities before the meeting, then compare. If everyone talks first, confident voices can anchor the room. If people forecast first, the group sees disagreement that would otherwise stay hidden.
A simple workplace version might ask three people to estimate the chance that a launch slips by a month. If the estimates are 20%, 45% and 70%, the value is not in averaging mechanically. The value is in asking what each person knows. The 70% forecaster may know about a dependency others missed; the 20% forecaster may know that a feared blocker has already been solved. The feedback loop improves because the group is not just exchanging opinions, but testing the assumptions underneath them.
Where formal scoring becomes too heavy
The Good Judgment Project’s scoring discipline is powerful, but everyday life does not need a full tournament dashboard. Brier scores are useful when there are many comparable forecasts, clear resolution criteria and enough time to keep records. They become cumbersome when decisions are rare, emotionally sensitive, hard to define, or too dependent on values rather than factual outcomes.
A useful boundary is this: score predictions when accuracy is the main issue; use reflective review when wisdom includes values, relationships or ethics. “Will this supplier deliver by Friday?” is forecastable. “Should I forgive this person?” contains predictions, but it is not reducible to a score. The Good Judgment lesson still helps, but indirectly: identify the factual expectations inside the decision without pretending the whole decision is a maths problem.
Formal scoring can also mislead if the questions are too easy or poorly chosen. A low Brier score on obvious predictions may not show much skill, while a bad score on a rare disrupted outcome may hide good reasoning. Good Judgment-style feedback works best when paired with a process review: What did I know? What did I miss? Was my confidence justified? What would I do differently next time? [PMC]pmc.ncbi.nlm.nih.govOpen source on nih.gov.
The everyday payoff
The practical value of the Good Judgment Project is that it makes judgement improvable. It turns confidence from a personality trait into a measurable habit. Instead of asking who sounds smartest, the feedback loop asks whose expectations survive contact with reality, who updates well, and who learns from misses without rewriting the past.
For improving thinking and analytical skills, that is the durable lesson. Good judgement is not just having better ideas in the moment. It is creating conditions in which reality can correct you often enough, clearly enough and safely enough that your future confidence becomes better calibrated than your past confidence.
Amazon book picks
Further Reading
Books and field guides related to What Forecasting Tournaments Teach Daily Decisions. Use these as the next step if you want deeper reading beyond the article.
Superforecasting
Directly explains the Good Judgment Project and transferable forecasting habits.
The Signal and the Noise
Expands on probabilistic thinking and improving real-world forecasts.
Expert Political Judgment
Provides the research foundations that led to the Good Judgment Project.
Thinking in Bets
Teaches probability-based decision habits applicable outside formal tournaments.
eBay marketplace picks
Marketplace Samples
Topic-anchored marketplace searches for visual, collectible, or second-hand items related to this page.
Endnotes
-
Source: iarpa.gov
Link: https://www.iarpa.gov/research-programs/aceSource snippet
ACEThe goal of the ACE Program is to dramatically enhance the accuracy, precision, and timeliness of intelligence forecasts for a br...
-
Source: web.stanford.edu
Link: https://web.stanford.edu/~knutson/jdm/mellers15.pdf -
Source: cambridge.org
Link: https://www.cambridge.org/core/journals/judgment-and-decision-making/article/developing-expert-political-judgment-the-impact-of-training-and-practice-on-judgmental-accuracy-in-geopolitical-forecasting-tournaments/123EB18425391D05FA6581FDBB3F309F -
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7333631/ -
Source: profiles.wustl.edu
Link: https://profiles.wustl.edu/en/publications/psychological-strategies-for-winning-a-geopolitical-forecasting-t/Source snippet
WashU Research ProfilesPsychological Strategies for Winning a Geopolitical...by B Mellers · 2014 · Cited by 434 — Forecasting is often v...
-
Source: gjopen.com
Link: https://www.gjopen.com/faqSource snippet
Good Judgment OpenFrequently Asked Questions (FAQ)Roughly, the Brier score indicates how far away from the truth your forecast was. The B...
-
Source: goodjudgment.com
Link: https://goodjudgment.com/resources/the-superforecasters-track-record/the-first-championship-season/ -
Source: goodjudgment.com
Link: https://goodjudgment.com/ -
Source: goodjudgment.com
Link: https://goodjudgment.com/resources/the-superforecasters-track-record/ -
Source: goodjudgment.com
Title: the science of superforecasting
Link: https://goodjudgment.com/about/the-science-of-superforecasting/ -
Source: goodjudgment.com
Link: https://goodjudgment.com/human-vs-ai-forecasts/ -
Source: goodjudgment.com
Link: https://goodjudgment.com/what-forecastbench-doesnt-measure/ -
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC9023363/ -
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/24659192/ -
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/25987508/ -
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC10189590/ -
Source: Wikipedia
Title: The Good Judgment Project
Link: https://en.wikipedia.org/wiki/The_Good_Judgment_Project -
Source: gjopen.com
Link: https://www.gjopen.com/snapcasts
Additional References
-
Source: psychologicalscience.org
Title: Association for Psychological Science Forecasting Tournaments
Link: https://www.psychologicalscience.org/journals/current-directions/0963721414534257/Source snippet
Association for Psychological ScienceForecasting Tournaments - Current Directions in...Forecasting tournaments are level-playing-field c...
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=_5YRFQmAxN0Source snippet
Superforecasting: The Art and Science of Prediction by Philip E. Tetlock...
-
Source: journals.sagepub.com
Link: https://journals.sagepub.com/doi/abs/10.1177/0956797614524255Source snippet
Sage JournalsPsychological Strategies for Winning a Geopolitical...by B Mellers · 2014 · Cited by 434 — Results showed that probability...
-
Source: youtube.com
Title: Why Intelligent People Are Wrong More Often Than Chimps | Philip Tetlock
Link: https://www.youtube.com/watch?v=qAKSd4eaSuYSource snippet
How to Predict the Future (According to Science)...
-
Source: youtube.com
Title: How to Predict the Future (According to Science)
Link: https://www.youtube.com/watch?v=W9sFZkYqvLoSource snippet
Why Predictions Fail | Use Probabilities Instead of Certainty...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/274992096_Forecasting_Tournaments_Tools_for_Increasing_Transparency_and_Improving_the_Quality_of_Debate -
Source: researchgate.net
Link: https://www.researchgate.net/publication/261034938_Psychological_Strategies_for_Winning_a_Geopolitical_Forecasting_Tournament -
Source: researchgate.net
Link: https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions -
Source: reddit.com
Link: https://www.reddit.com/r/slatestarcodex/comments/p85tc1/a_failure_but_of_prediction/ -
Source: dokumen.pub
Link: https://dokumen.pub/expert-political-judgment-how-good-is-it-how-can-we-know-new-edition-newnbsped-9781400888818.html
Topic Tree