Within Alternatives
The Test That Breaks A Tie
The best evidence check asks what one explanation would make likely and a rival would make surprising.
On this page
- Turning explanations into expected observations
- Choosing checks that discriminate rather than confirm
- Using predictions before gathering new evidence
Page outline Jump by section
Introduction
When two explanations both fit the facts you already know, asking whether the evidence is consistent with your favourite explanation is not enough. The more useful question is: what would each explanation predict that the other would find surprising? A prediction test is designed to answer exactly that question. Instead of collecting more evidence at random, you identify observations that would strongly favour one explanation over its rivals.
This approach improves analytical thinking because it forces explanations to compete on what they expect to happen, not merely on how well they explain the past. It also reduces the risk of confirmation bias by making you look for evidence that could genuinely change your mind rather than evidence that merely fits a preferred story. Research in psychology, philosophy of science and intelligence analysis consistently finds that evidence is most valuable when it discriminates between competing hypotheses rather than simply accumulating support for one of them. [UC San Diego Pages+2Strathprints]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
Turning explanations into expected observations
A useful explanation is more than a story about what has already happened. It also implies what else should be true if that explanation is correct.
Suppose sales have fallen.
- Explanation A: customers dislike the product.
- Explanation B: customers cannot find the new purchasing process.
Both explain declining sales, but they predict different observations. If the product is the problem, customer satisfaction scores, product reviews and repeat purchase rates should also decline. If the purchasing process is the problem, customers may abandon transactions while product ratings remain stable.
The key step is translating each explanation into concrete expectations.
Rather than asking:
“Does the evidence fit my explanation?”
ask:
“If this explanation were true, what else should I expect to observe?”
Then ask the same question for every serious alternative.
A good prediction test therefore has three features:
- it identifies an observation that has not yet been used to justify the explanation;
- it specifies what each explanation predicts before looking;
- it allows the competing explanations to make different predictions.
This reflects a long-standing principle in scientific reasoning: explanations gain credibility not merely by accommodating known facts but by successfully predicting observations that were not built into the explanation after the fact. [eScholarship]escholarship.orgUC MercedMarch 11, 2025 — by LE Strittmatter · 2023 — Theories can be designed to predict novel evidence or to accommodate kn…
Choose tests that discriminate rather than merely confirm
Not all evidence is equally informative.
Imagine two explanations for why a website’s traffic suddenly increased.
- A marketing campaign attracted new visitors.
- A search engine algorithm changed rankings.
Checking whether traffic increased tells you almost nothing—both explanations predicted that.
Checking whether visitors arrived primarily through paid advertisements or through organic search, however, separates the explanations.
The most valuable evidence is therefore diagnostic. It is much more likely under one explanation than another.
A practical question is:
“Which observation would make me substantially more confident in one explanation while making the rival less likely?”
This differs from collecting “more evidence” in general.
Weak confirmation:
- finding another dissatisfied customer when every explanation already expected some complaints.
Strong discrimination:
- discovering that dissatisfaction appears only among customers who experienced a particular onboarding step.
The second observation changes the relative balance between explanations. The first mostly increases confidence in all of them simultaneously.
Structured analytical methods such as the Analysis of Competing Hypotheses (ACH) explicitly encourage analysts to identify evidence that distinguishes hypotheses instead of simply counting supporting facts. They also emphasise paying particular attention to inconsistent evidence because disagreement often carries more information than agreement. [Strathprints+2Futuribles]strathprints.strath.ac.ukDhami etal ACP 2019 The analysis of competing hypotheses in intelligencebcIn ACH, the credibility and…
Make predictions before gathering new evidence
Prediction tests work best when expectations are written down before collecting new information.
Once evidence has been seen, people naturally reinterpret their explanations to accommodate it. This makes almost any explanation appear more successful than it really is.
Writing predictions first limits this flexibility.
A simple format is:
ExplanationIf true, I expect…I would be surprised if…AObservation XObservation YBObservation YObservation X
The wording matters.
Strong predictions are specific enough that someone else could recognise success or failure.
Compare:
- “People will react positively.”
with:
- “At least 70% of first-time users will complete onboarding without assistance.”
The second creates a meaningful opportunity for explanations to succeed or fail.
Philosophers of science have long distinguished theories that merely accommodate existing evidence from those that successfully predict new observations. Although prediction is not automatically superior in every circumstance, successful novel predictions often provide stronger support because they were not constructed after the outcome became known. [eScholarship]escholarship.orgUC MercedMarch 11, 2025 — by LE Strittmatter · 2023 — Theories can be designed to predict novel evidence or to accommodate kn…
Look for asymmetric predictions
The best prediction tests are often asymmetric.
One explanation may regard an observation as almost inevitable while another considers it highly unlikely.
For example:
- Explanation A: The machine failed because a component overheated.
- Explanation B: The machine failed because software settings were incorrect.
Both explain the shutdown.
However:
- discovering heat damage strongly favours Explanation A;
- discovering identical failures across multiple machines immediately after a software update strongly favours Explanation B.
Notice that these are not merely additional facts. They carry unequal weight because they are expected by one explanation far more than the other.
Thinking in terms of asymmetry also discourages vague explanations that can explain almost anything. If an explanation predicts every possible outcome equally well, it predicts nothing useful.
Beware of explanations that can absorb every result
Some explanations appear persuasive because they can be adjusted after every observation.
For example:
“If customers complain, they dislike the product.”
“If customers do not complain, they quietly dislike the product.”
Such explanations never risk being wrong.
Prediction tests expose this weakness by insisting on observations that would genuinely count against the explanation.
Ask:
- What finding would make me reduce confidence?
- What result would clearly support the rival explanation?
- Am I allowing my explanation to change after every new observation?
If no imaginable evidence could distinguish between competing explanations, then the disagreement may be more about storytelling than about evidence.
This idea also appears in discussions of confirmation bias. Simply finding additional compatible evidence is often much less informative than deliberately searching for observations that could differentiate competing hypotheses. [UC San Diego Pages]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
A practical workflow for prediction testing
When several plausible explanations remain alive, a structured sequence helps prevent premature commitment.
- State the competing explanations clearly. Each should describe a distinct mechanism rather than merely restating the outcome.
- List unique predictions. Write down observations that each explanation specifically expects.
- Identify the most diagnostic test. Choose evidence that would favour one explanation much more than another.
- Record predictions before collecting evidence. This reduces hindsight and motivated reinterpretation.
- Update comparatively rather than absolutely. Ask which explanation became stronger relative to its competitors rather than whether one explanation still seems possible.
This approach mirrors Bayesian thinking, where evidence is evaluated by how much more expected it is under one hypothesis than another, rather than by whether it is merely compatible with a preferred explanation. [PMC+2strevens.org]pmc.ncbi.nlm.nih.govPMCA Tutorial on Conducting and Interpreting a Bayesianby HE Malone · 2025 · Cited by 3 — Researchers formulate a hypothesis and collect data to test that hypothesis. Bayesian analysis focu…
Why this habit improves analytical thinking
Prediction tests change the role of evidence. Instead of serving as decoration for an already accepted story, evidence becomes a way of forcing competing explanations to make risky commitments.
The habit produces several benefits:
- it reduces confirmation bias by directing attention towards potentially disconfirming observations;
- it encourages precise rather than vague explanations;
- it makes evidence collection more efficient because diagnostic observations receive priority;
- it increases the chance that genuine surprises will improve understanding instead of being explained away.
Most importantly, prediction tests transform uncertainty from a weakness into a productive stage of reasoning. When several explanations remain plausible, the objective is not to defend a favourite but to discover which explanation makes the most accurate predictions about observations that have not yet been seen. That is the point at which competing explanations stop being stories and begin functioning as genuine tests of understanding. [Wikipedia+3UC San Diego Pages+3Strathprints]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
Amazon book picks
Further Reading
Books and field guides related to The Test That Breaks A Tie. Use these as the next step if you want deeper reading beyond the article.
Superforecasting
Directly addresses prediction-based evaluation of competing explanations.
Psychology of Intelligence Analysis
Explains discriminating evidence and rival hypotheses.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11169332/Source snippet
of task structure and confirmation bias in alternative...by MK Dhami · 2024 · Cited by 2 — We empirically examined the effectiveness of...
-
Source: escholarship.org
Link: https://escholarship.org/content/qt4b8349fn/qt4b8349fn.pdfSource snippet
UC MercedMarch 11, 2025 — by LE Strittmatter · 2023 — Theories can be designed to predict novel evidence or to accommodate kn...
Published: March 11, 2025
-
Source: Wikipedia
Title: Strong inference
Link: https://en.wikipedia.org/wiki/Strong_inference -
Source: futuribles.com
Link: https://www.futuribles.com/wp-content/uploads/related-documents/analysis-of-competing-hypotheses.pdf?postId=73706Source snippet
Analysis of Competing HypothesesAnalysis of Competing Hypotheses (ACH) is an intelligence analysis method based on evaluating plausible a...
-
Source: strevens.org
Link: https://www.strevens.org/bct/BCT.pdfSource snippet
Notes on Bayesian Confirmation Theoryby M Strevens · 2017 · Cited by 51 — The Bayesian apparatus, it seems, is a complete guide to how yo...
-
Source: Wikipedia
Title: Confirmation bias
Link: https://en.wikipedia.org/wiki/Confirmation_biasSource snippet
Confirmation biasConfirmation bias is the tendency to search for, interpret, favor and recall information in a way that confirms or su...
-
Source: youtube.com
Title: Intelligence Analysis Skills: Analysis of Competing Hypotheses (Part 1)
Link: https://www.youtube.com/watch?v=J_eDCBf7R2ISource snippet
Analysis of Competing Hypotheses...
-
Source: youtube.com
Title: Analysis of Competing Hypotheses
Link: https://www.youtube.com/watch?v=t6GEvRYMIxsSource snippet
(ACH): A Structured Analytic Technique (SAT) for FinCrime...
-
Source: pages.ucsd.edu
Title: UC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in Many
Link: https://pages.ucsd.edu/~mckenzie/nickersonConfirmationBias.pdfSource snippet
UC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many...October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi...
Published: October 6, 2004
-
Source: strathprints.strath.ac.uk
Title: Dhami etal ACP 2019 The analysis of competing hypotheses in intelligence
Link: https://strathprints.strath.ac.uk/69049/1/Dhami_etal_ACP_2019_The_analysis_of_competing_hypotheses_in_intelligence.pdfSource snippet
bcIn ACH, the credibility and...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCA Tutorial on Conducting and Interpreting a Bayesian
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC12994668/Source snippet
by HE Malone · 2025 · Cited by 3 — Researchers formulate a hypothesis and collect data to test that hypothesis. Bayesian analysis focu...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8111937/Source snippet
predictions arise from contradictions - PMC - NIHby I Yanai · 2021 · Cited by 9 — Confirmation bias leads scientists to dismiss or misint...
Additional References
-
Source: nyu-cdsc.github.io
Link: https://nyu-cdsc.github.io/learningr/assets/kruschke_bayesian_in_R.pdf -
Source: researchgate.net
Link: https://www.researchgate.net/publication/331029513_A_Tutorial_on_Testing_Hypotheses_Using_the_Bayes_FactorSource snippet
A Tutorial on Testing Hypotheses Using the Bayes FactorIn this tutorial it is elaborated how researchers can use the Bayes factor for the...
-
Source: sosintel.co.uk
Link: https://sosintel.co.uk/mastering-the-analysis-of-competing-hypotheses-ach-a-practical-framework-for-clear-thinking/Source snippet
Mastering the Analysis of Competing Hypotheses (ACH)20 Jun 2025 — The Analysis of Competing Hypotheses (ACH) is a structured method desig...
-
Source: repository.mdx.ac.uk
Title: evidence from various dif- ferent sources relates to two competing hypotheses
Link: https://repository.mdx.ac.uk/download/56199b865755f7ffe0f2e697a221094d9723cc68c1db2314d90d96fcdd117016/1342655/s41235-024-00560-y.pdfSource snippet
of task structure and confirmation bias in alternative...by MK Dhami · 2024 · Cited by 2 — Similarly, early research has shown that a 'c...
-
Source: thechangelab.stanford.edu
Title: the basics of the bayesian approach an introductory tutorial
Link: https://thechangelab.stanford.edu/tutorials/bayesian-methods/the-basics-of-the-bayesian-approach-an-introductory-tutorial/Source snippet
Basics of the Bayesian Approach: An Introductory TutorialIn this tutorial, we begin laying the groundwork for understanding the Bayesian...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/220480585_Seeking_Confirmation_Is_Rational_for_Deterministic_HypothesesSource snippet
hen those hypotheses are deterministic, each making a single prediction about...Read more...
-
Source: uu.nl
Link: https://www.uu.nl/en/research/the-bettr-project/about/bayesian-and-non-bayesian-evaluation-of-informative-hypothesesSource snippet
They allow us to calculate different model comparison metrics (e.g....Read more...
-
Source: academia.edu
Link: https://www.academia.edu/47828252/The_Analysis_of_Competing_Hypotheses_in_Intelligence_AnalysisSource snippet
esigned to reduce "confirmation bias." Fifty intelligence analysts were randomly...
-
Source: kicj.re.kr
Link: https://www.kicj.re.kr/boardDownload.es?bid=0034&list_no=12219&seq=1Source snippet
CH) efficacy. The technique was developed by the Central Intelligence...
-
Source: youtu.be
Link: https://youtu.be/dd5KU9VzwWoSource snippet
"Gaussian Distribution - ML Snippets: [https://youtu.be/ySrK1DCQYA4](https://youtu.be/ySrK1DCQYA4) Statistical Moments: Mean, Variation, Skewness, Kurtosis: [https://youtu..."](https://youtu...")...
Topic Tree