Within Confirmation Bias
The Test Your Favorite Story Needs
A useful test does not just ask whether your favorite explanation can survive; it asks what would separate it from a serious rival.
On this page
- What positive testing gets right
- Where positive testing misleads decisions
- How to compare rival explanations
Page outline Jump by section
Introduction
Positive testing is one of the most misunderstood ideas in research on confirmation bias. It is often described as a flaw in reasoning, but the evidence shows something more subtle. People naturally test a hypothesis by looking for situations where they expect it to be true. This “positive test strategy” is often efficient because it focuses attention on cases that are likely to be informative. The problem arises when that strategy becomes the only strategy, especially in personal decisions where a preferred explanation has emotional appeal. Instead of asking whether a favourite story can gather supporting evidence, good reasoning asks what evidence would distinguish it from the strongest competing explanation. That shift—from supporting one story to comparing rival stories—is one of the most effective ways to reduce confirmation bias.[UC San Diego Pages]pages.ucsd.eduUC San Diego PagesConfirmation, Discontinuation, and Information in…October 7, 2004 — Next, we show how the positive test strategy pro…
What positive testing gets right
Early discussions of confirmation bias often gave the impression that people irrationally searched only for confirming evidence. Joshua Klayman and Young-Won Ha challenged that interpretation by arguing that many experiments were better explained by a general positive test strategy than by a simple desire to confirm beliefs. Rather than asking questions that could produce either answer with equal probability, people tend to examine situations where the hypothesised property is expected to appear.[UC San Diego Pages]pages.ucsd.eduUC San Diego PagesConfirmation, Discontinuation, and Information in…October 7, 2004 — Next, we show how the positive test strategy pro…
This strategy is not inherently irrational. In many everyday situations it is an efficient heuristic—a mental shortcut that reduces the enormous number of possible tests. If you suspect that a plant dies because it receives too little water, looking at under-watered plants is often more informative than randomly examining every plant regardless of watering. Similarly, if you believe a particular study method improves exam performance, comparing people who actually used that method can be a sensible place to start.
Klayman and Ha argued that under common real-world conditions, positive testing often provides useful information because the hypotheses people investigate are usually quite specific rather than broad. In those circumstances, checking where the predicted effect should appear is often a reasonable first move. The mistake is assuming that a useful heuristic is automatically a complete method for reaching sound conclusions.[UC San Diego Pages]pages.ucsd.eduUC San Diego PagesConfirmation, Discontinuation, and Information in…October 7, 2004 — Next, we show how the positive test strategy pro…
Where positive testing misleads decisions
The weakness of positive testing appears when several plausible explanations could account for the same observations. Supporting evidence for one explanation is often perfectly compatible with another.
Suppose someone believes:
“My manager criticised my presentation because they dislike me.”
Looking only for confirming observations may reveal that the manager was curt during the meeting or failed to smile afterwards. Those observations fit the hypothesis. However, they may also fit alternative explanations:
- The manager was under unusual time pressure.
- The presentation genuinely contained weaknesses.
- The manager behaves similarly with everyone.
- The organisation has recently tightened performance expectations.
Each additional confirming observation may increase confidence without actually increasing the ability to distinguish between these possibilities.
This illustrates an important principle in hypothesis testing: evidence has greatest value when it is diagnostic, meaning it changes the balance between competing explanations rather than merely fitting one of them. Many observations support multiple stories simultaneously. They feel persuasive because they are consistent with the preferred explanation, not because they uniquely identify it.[UC San Diego Pages]pages.ucsd.eduUC San Diego PagesConfirmation, Discontinuation, and Information in…October 7, 2004 — Next, we show how the positive test strategy pro…
Personal decisions are especially vulnerable because preferred explanations often satisfy emotional needs. A hopeful investor wants a company to be undervalued. Someone considering a relationship wants awkward behaviour to be temporary. Someone choosing between jobs wants reassuring signs that justify an exciting opportunity. Positive testing then becomes selective information gathering rather than genuine evaluation.
The rival-explanation test
The rival-explanation test asks a different question:
What evidence would separate my preferred explanation from the strongest alternative?
This shifts attention from accumulating supportive evidence to comparing competing models of reality.
Instead of asking:
- “Can I find evidence that supports my view?”
you ask:
- “What would I expect to observe if my explanation were wrong but the best alternative were correct?”
The distinction is subtle but powerful.
Imagine deciding whether a freelance business is succeeding because of excellent marketing.
A positive test looks for:[pages.ucsd.edu]pages.ucsd.eduUC San Diego PagesConfirmation, Discontinuation, and Information in…October 7, 2004 — Next, we show how the positive test strategy pro…
- increasing enquiries,
- positive customer comments,
- growing website traffic.
A rival-explanation test asks whether these same observations could equally be explained by:
- seasonal demand,
- temporary market shortages,
- unusually favourable economic conditions,
- referral effects independent of marketing.
The next useful question becomes:
“What evidence would differ if marketing really were the main cause?”
Perhaps new customers could be asked how they found the business. Conversion rates could be compared before and after specific campaigns. Different marketing channels could be tested separately. Those observations discriminate between explanations rather than merely supporting one.
Why comparing rivals improves judgement
Psychological research suggests that people often generate and evaluate a single working hypothesis rather than several simultaneously. Once attention centres on one explanation, information search naturally follows that direction. This creates an illusion that evidence is becoming increasingly convincing when, in reality, alternatives have simply received less attention.[UC San Diego Pages]pages.ucsd.eduUC San Diego PagesConfirmation, Discontinuation, and Information in…October 7, 2004 — Next, we show how the positive test strategy pro…
Comparing rival explanations improves judgement for several reasons.
It changes the standard of evidence. Instead of asking whether evidence is consistent with a favourite explanation, you ask whether it is more consistent with one explanation than another.
It exposes missing information. Rival explanations often reveal variables that have not yet been measured or observed.
It reduces motivated interpretation. When several explanations are evaluated together, it becomes harder to excuse weaknesses in only one preferred account while demanding unusually high standards from alternatives.
It encourages prediction rather than storytelling. Competing explanations often generate different expectations about future events. Predictions can later be checked against reality.
How to compare rival explanations
The rival-explanation test does not require formal statistical training. It can be incorporated into ordinary decisions through a structured comparison.
- State the preferred explanation clearly. Avoid vague impressions such as “this feels right.”
- Generate at least one serious alternative. The goal is not to invent absurd possibilities but plausible competitors.
- Ask what each explanation predicts. Different explanations should imply different observations, behaviours or future outcomes.
- Seek diagnostic evidence. Prioritise information that distinguishes between explanations rather than information that merely fits one.
- Be willing to update confidence gradually. Evidence rarely proves one explanation absolutely correct. Instead, it should shift confidence relative to alternatives.
For example, someone deciding whether persistent fatigue reflects excessive workload should compare that explanation with alternatives such as poor sleep habits, medication side effects or an underlying medical condition. The most useful evidence is not another exhausting week at work—which fits several explanations—but information that discriminates among them, such as changes after improving sleep, reviewing medication or obtaining appropriate medical assessment.
A useful habit for personal decisions
The practical lesson from research is not to abandon positive testing altogether. Positive testing is often an efficient starting point because it quickly explores the implications of a working hypothesis. Problems arise when reasoning stops there.
A stronger habit is to treat every important personal decision as a competition between explanations rather than a search for support for the first attractive story. Before committing to a conclusion, ask:
- What is the strongest rival explanation?
- What observation would favour that rival over my preferred story?
- Have I actively looked for that observation?
Those questions transform evidence gathering from an exercise in reassurance into an exercise in discrimination. Instead of merely asking whether your favourite explanation can survive, they ask whether it deserves to win against its most credible competitor—a much tougher and far more reliable test of sound judgement.[UC San Diego Pages+2Sage Journals]pages.ucsd.eduUC San Diego PagesConfirmation, Discontinuation, and Information in…October 7, 2004 — Next, we show how the positive test strategy pro…
Amazon book picks
Further Reading
Books and field guides related to The Test Your Favorite Story Needs. Use these as the next step if you want deeper reading beyond the article.
The Scout Mindset
Directly advocates evaluating rival hypotheses instead of defending favored beliefs.
Thinking, Fast and Slow
Explains confirmation bias, heuristics, and why comparing alternative explanations improves judgment.
Superforecasting
Shows practical methods for weighing competing explanations and updating beliefs using evidence.
The Demon-Haunted World
Rating: 4.5/5 from 43 Google Books ratings
Promotes skeptical reasoning and testing competing explanations rather than accepting attractive stories.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Top pick
critical thinking card game
Open a targeted eBay search for items related to this topic.
Browse eBay
Related search
critical thinking game
Open a targeted eBay search for items related to this topic.
Browse eBay
Related search
your memorabilia
Open a targeted eBay search for items related to this topic.
Browse eBayEndnotes
1.
Source: pages.ucsd.edu
Link:https://pages.ucsd.edu/~mckenzie/KlaymanHaPsychReview1987.pdf
Source snippet
UC San Diego PagesConfirmation, Discontinuation, and Information in...October 7, 2004 — Next, we show how the positive test strategy pro...
Published: October 7, 2004
2.
Source: journals.sagepub.com
Title: 1089 2680.2.2.175
Link:https://journals.sagepub.com/doi/10.1037/1089-2680.2.2.175
Source snippet
Sage JournalsConfirmation Bias: A Ubiquitous Phenomenon in Many...Klayman and Ha (1987)argued that the positive-test strategy is sometim...
3.
Source: Wikipedia
Title: Confirmation bias
Link:https://en.wikipedia.org/wiki/Confirmation_bias
Source snippet
Confirmation biasConfirmation bias is the tendency to search for, interpret, favor and [recall]({{ 'recall/' | relative_url }}) information in a way that confirms or su...
Additional References
4.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC11561113/
Source snippet
by V Berthet · 2024 · Cited by 26 — Abstract. When they are asked to test a given hypothesis, individuals tend to be biased towards co...
5.
Source: ius.uzh.ch
Title: Simon and Read (203) Toward a General Framework of Biased Reasoning
Link:https://www.ius.uzh.ch/dam/jcr%3Ab7b5fdad-e48d-450e-83d2-32274fc88325/Simon%20and%20Read%20%28203%29%20-%20Toward%20a%20General%20Framework%20of%20Biased%20Reasoning.pdf
Source snippet
a General Framework of Biased Reasoningby D Simon · 2023 · Cited by 42 — Other biases have been explained as related to nonsystematic evi...
6.
Source: youtube.com
Title: CONFIRMATION BIAS: Why we only ask “yes” questions?
Link:https://www.youtube.com/watch?v=dVkPXwA46Ek
Source snippet
Alternative explanations rival hypothesis critical thinking decision making DECISION Making - Meaning & Stages involved in Decision Makin...
7.
Source: documents.theblackvault.com
Title: intellipedia confirmationbias
Link:https://documents.theblackvault.com/documents/intellipedia/intellipedia-confirmationbias.pdf
Source snippet
bias25 May 2018 —... test alternative hypotheses in parallel[68l Another heuristic is the positive test strategy identified by Klayman a...
Published: May 2018
8.
Source: researchgate.net
Link:https://www.researchgate.net/publication/380309922_A_common_factor_underlying_confirmation_bias_in_hypothesis_testing_tasks
Source snippet
However, little is known about individual...Read more...
9.
Source: drcharlesmrusso.substack.com
Title: disconfirmatory search and belief
Link:https://drcharlesmrusso.substack.com/p/disconfirmatory-search-and-belief
Source snippet
Search and Belief RevisionKlayman and Ha (1987) complicated the matter by showing that people often use a “positive test strategy,” seeki...
10.
Source: youtube.com
Title: Karl Popper, Science, & Pseudoscience: Crash Course Philosophy #8
Link:https://www.youtube.com/watch?v=-X8Xfl0JdTQ
Source snippet
CONFIRMATION BIAS: Why we only ask "yes" questions?...
11.
Source: youtube.com
Title: Six Principles of Scientific Thinking in Psychology
Link:https://www.youtube.com/watch?v=ysb3eo8xuUc
Source snippet
Karl Popper, Science, & Pseudoscience: Crash Course Philosophy #8...
12.
Source: researchsquare.com
Link:https://www.researchsquare.com/article/rs-4318265/v1.pdf
Source snippet
A common factor underlying con rmation bias in hypothesis...by V Berthet · 2024 — Indeed, prior studies suggested that the 2-4-6 and int...
13.
Source: youtube.com
Title: Karl Popper’s Falsification
Link:https://www.youtube.com/watch?v=wf-sGqBsWv4
Source snippet
Six Principles of Scientific Thinking in Psychology...
Topic Tree