Within Alternatives
When Evidence Fits Too Many Stories
Evidence that fits your favorite story may be weak if the same fact would also fit serious rivals.
On this page
- Why compatible evidence feels stronger than it is
- How to spot facts shared by several explanations
- Examples from symptoms, sales drops, and performance reviews
Page outline Jump by section
Introduction
A fact is only strong evidence if it helps you choose between competing explanations. If the same observation would be expected under several plausible explanations, then it confirms very little on its own. This is one of the easiest mistakes to make in everyday reasoning: treating a fact that is merely compatible with your preferred story as though it favours that story over its rivals. Better judgement depends on asking a different question. Instead of asking, “Does this fit my explanation?”, ask, “Would I be any less surprised to see this fact if another serious explanation were true?” That shift turns evidence from a simple match into a genuine test.
Why compatible evidence feels stronger than it is
People naturally search for facts that agree with their current thinking. The problem is not that confirming evidence is worthless, but that it often receives too much weight before we ask whether it is distinctive. Research on confirmation bias shows that people tend to seek, notice and interpret information in ways that support an existing hypothesis while giving less attention to competing possibilities. [UC San Diego Pages]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
Imagine hearing a strange noise from your car. Discovering that the engine warning light is on may seem to support the idea of engine trouble. Yet the same warning light could also appear because of an electrical fault, a failing sensor or a transmission problem. The observation is compatible with several explanations. It therefore narrows the field only slightly.
The same logic applies in scientific reasoning. A prediction that comes true is impressive only if rival theories would not have predicted the same outcome. If every serious explanation expected the observation, then the observation does not distinguish between them.
This idea appears in philosophy of science and Bayesian approaches to evidence, where evidence is judged by how much it changes the relative credibility of competing hypotheses rather than by whether it merely fits one of them. [Wikipedia]WikipediaBayesian epistemologyBayesian epistemology
What makes evidence genuinely informative?
The most valuable evidence is diagnostic: it is much more likely if one explanation is true than if another is.
A useful way to think about any observation is to ask:
- Would I expect to see this under my preferred explanation?
- Would I also expect to see it under the main alternatives?
- If both answers are “yes”, how much does this observation really change anything?
The more similar those answers become, the weaker the evidence becomes.
For example:
ObservationPossible explanationsHow informative is it?A patient has a fever.Viral infection, bacterial infection, autoimmune disease and many others.Weak by itself because many conditions cause fever.A company’s sales fall.Poor product, recession, stronger competitors, supply problems or pricing changes.Weak until more specific evidence appears.An employee seems less engaged.Burnout, illness, family stress, lack of challenge, conflict with colleagues or poor management.Weak without additional distinguishing facts.
None of these observations is useless. They simply cannot identify the correct explanation on their own.
How to spot facts shared by several explanations
One practical habit is to separate matching evidence from discriminating evidence.
Matching evidence answers:
“Could this happen if my explanation were correct?”
Discriminating evidence answers:
“Is this observation noticeably more likely under my explanation than under the alternatives?”
Many people stop after the first question because finding a match feels satisfying. But matching alone rarely settles anything.
A helpful exercise is to list two or three realistic rival explanations before examining the evidence. Then take each important fact and ask whether every explanation predicts it.
If every column receives a tick, the evidence is shared rather than distinguishing.
For example, suppose website traffic suddenly falls.
ExplanationPredicts lower traffic?Search engine ranking droppedYesAdvertising budget was reducedYesSeasonal demand declinedYes
Traffic alone tells you almost nothing.
Now consider another observation: search impressions remain steady while paid advertising expenditure was cut by 70%.
That second observation separates the explanations far better because it is not equally expected under all of them.
Examples in everyday judgement
Symptoms and medical diagnosis
Many symptoms are deliberately treated as non-specific because they occur across numerous illnesses. Coughs, headaches and fatigue often indicate that something is wrong but do not identify what.
Good diagnosis therefore depends on evidence that separates possibilities rather than merely confirms one. Doctors continually revise diagnoses by asking which additional signs, laboratory results or imaging findings would be expected under competing diseases. Prematurely settling on one explanation risks overlooking alternatives that explain the same early symptoms equally well. [UC San Diego Pages]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
Sales declines
A manager who believes product quality has deteriorated may collect customer complaints as confirmation.
But customer complaints are compatible with many explanations:
- higher customer expectations
- delayed deliveries
- pricing changes
- increased competition
- confusing marketing
- genuine quality problems
The complaints become informative only when examined alongside evidence that distinguishes between these possibilities, such as defect rates, competitor comparisons or customer interviews focused on specific causes.
Performance reviews
Suppose a manager thinks an employee has become less motivated.
Evidence such as missed deadlines or reduced participation certainly fits that explanation.
However, the same behaviour could arise from:
- unclear priorities
- excessive workload
- health problems
- poor team coordination
- lack of required training
Until evidence uniquely favours one explanation, the observation should remain provisional rather than decisive.
Why shared evidence creates overconfidence
Compatible evidence feels persuasive because humans naturally ask whether observations are consistent with what they already believe.
Psychologist Peter Wason’s classic work on hypothesis testing showed that people often search for examples that fit a rule instead of looking for cases that could distinguish between competing rules. Later research argued that people frequently employ a “positive test strategy”: checking whether expected evidence appears rather than asking whether alternative explanations would produce the same observation. [Wikipedia]WikipediaConfirmation biasConfirmation bias
This habit creates a subtle illusion.
As each compatible fact accumulates, confidence grows:
- “The customer complained.”
- “Sales declined.”
- “The employee looked frustrated.”
Individually, each observation may fit the preferred explanation.
Collectively, they may still fail to distinguish it from several equally plausible rivals.
The result is confidence without proportionate evidence.
Practical questions that improve judgement
Whenever evidence seems to support your favourite explanation, pause and ask:
- What else would predict this same observation?
- Which explanation expected this fact most strongly?
- What observation would clearly separate these explanations?
- Am I collecting evidence that merely agrees with my idea, or evidence that would actually make me reject it?
These questions shift attention from simple confirmation towards comparison.
The key lesson
Evidence is not valuable simply because it fits a story. Its value depends on whether it helps choose between competing stories.
Many observations are shared across several plausible explanations. Treating such observations as decisive leads to misplaced confidence, premature conclusions and poor decisions. Strong reasoning comes from seeking evidence that discriminates rather than merely agrees. The more an observation is expected under one explanation and not under serious alternatives, the more informative it becomes. Until then, the safest conclusion is often that the evidence has narrowed the possibilities less than it first appeared.
Amazon book picks
Further Reading
Books and field guides related to When Evidence Fits Too Many Stories. Use these as the next step if you want deeper reading beyond the article.
Thinking, Fast and Slow
Explains confirmation bias and overvaluing compatible evidence.
Psychology of Intelligence Analysis
Discusses diagnostic versus merely compatible evidence.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: Wikipedia
Title: Bayesian epistemology
Link: https://en.wikipedia.org/wiki/Bayesian_epistemology -
Source: Wikipedia
Title: Evidence under Bayes’ theorem
Link: https://en.wikipedia.org/wiki/Evidence_under_Bayes%27_theorem -
Source: Wikipedia
Title: Confirmation bias
Link: https://en.wikipedia.org/wiki/Confirmation_bias -
Source: pages.ucsd.edu
Title: UC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in Many
Link: https://pages.ucsd.edu/~mckenzie/nickersonConfirmationBias.pdfSource snippet
UC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many...October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi...
Published: October 6, 2004
Additional References
-
Source: thedecisionlab.com
Link: https://thedecisionlab.com/biases/confirmation-biasSource snippet
Confirmation BiasConfirmation bias describes our underlying tendency to notice, focus on, and provide greater credence to evidence that f...
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCA confirmation bias in perceptual decision-making due
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8659691/Source snippet
by RD Lange · 2021 · Cited by 81 — The confirmation bias, in which new evidence is given more weight when it agrees with existing beli...
-
Source: arnoldkling.substack.com
Title: confirmation bias a bayesian interpretation
Link: https://arnoldkling.substack.com/p/confirmation-bias-a-bayesian-interpretationSource snippet
Bias: A Bayesian InterpretationIn an example I like to use, during a routine checkup a doctor presented me with evidence that there was m...
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=wo3xpigIjtsSource snippet
gnitive bias and why it causes us to select information based...
-
Source: pdxscholar.library.pdx.edu
Link: https://pdxscholar.library.pdx.edu/cgi/viewcontent.cgi?article=1127&context=open_access_etdsSource snippet
Bias and Related Errors - PDXScholarby GL Borthwick · 2010 · Cited by 3 — Research on search and integration biases that can result in co...
-
Source: youtube.com
Title: What is [Critical Thinking]({{ ‘critical-skills/’ | relative_url }})?
Link: https://www.youtube.com/watch?v=HnJ1bqXUnIMSource snippet
The power of Bayesian reasoning | BBC Ideas - YouTube The power of Bayesian reasoning | BBC Ideas - YouTube...
-
Source: youtube.com
Title: The power of Bayesian reasoning | BBC Ideas
Link: https://www.youtube.com/watch?v=1LzdESG6-2ESource snippet
Analysis of Competing Hypotheses (ACH): Finding Plausible Answers...
-
Source: wrap.warwick.ac.uk
Title: WRAP confirmation bias emerges approximation Bayesian reasoning 2024
Link: https://wrap.warwick.ac.uk/id/eprint/181714/7/WRAP-confirmation-bias-emerges-approximation-Bayesian-reasoning-2024.pdfSource snippet
We show that confirmation...Read more...
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=Y-J0FYOQRMYSource snippet
What is Critical Thinking?...
-
Source: youtube.com
Title: Analysis of Competing Hypotheses
Link: https://www.youtube.com/watch?v=t6GEvRYMIxsSource snippet
(ACH): A Structured Analytic Technique (SAT) for FinCrime...
Topic Tree