Within Alternatives
The Search That Confirms Too Easily
Searching where your favorite explanation expects evidence can help, but it can also hide rival explanations.
On this page
- Why looking for expected cases can be useful
- When confirming searches become misleading
- How to add rival focused checks without overcorrecting
Page outline Jump by section
Introduction
When people test an idea, they rarely begin by asking, “What would prove me wrong?” More often, they look where their current explanation predicts they will find supporting evidence. Psychologists call this a positive test strategy: testing a hypothesis by examining cases where it is expected to be true. This is not inherently irrational. In many everyday situations it is an efficient way to gather information. The problem arises when the search never leaves the territory favoured by the initial story. If rival explanations would predict the same observations, a stream of confirming evidence may increase confidence without actually distinguishing between competing accounts. Research on hypothesis testing shows that the quality of thinking depends less on finding confirming examples than on asking whether the evidence would look different if another plausible explanation were true. [UC San Diego Pages]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
Why looking for expected cases can be useful
Positive testing has an intuitive appeal because hypotheses make predictions. If you suspect a plant is dying because it is not getting enough light, looking at the leaves and measuring light exposure is sensible. If you think a software bug appears only under heavy load, reproducing those conditions is a practical first step.
Joshua Klayman and Young-Won Ha argued that this tendency should not automatically be labelled irrational. They proposed that people often use a positive test strategy because it is a generally useful heuristic—a mental shortcut that frequently identifies informative evidence while requiring relatively little effort. Whether it succeeds depends on the structure of the problem. In many real-world situations, examining where an effect is expected genuinely provides the quickest route to learning something new. [Wikipedia]WikipediaConfirmation biasConfirmation bias
The key insight is that positive does not mean biased. A positive test can be highly diagnostic if only one explanation predicts the observation. For example:
- If only one machine in a factory could produce a particular defect pattern, inspecting products from that machine is an efficient positive test.
- If only one disease produces a distinctive laboratory marker, checking for that marker can rapidly narrow the diagnosis.
In these situations, searching where the hypothesis predicts evidence is exactly what good investigation requires.
When confirming searches become misleading
The difficulty appears when several explanations predict the same evidence.
Imagine a manager who believes falling productivity is caused by poor motivation. They interview employees already known to be disengaged. Unsurprisingly, they hear complaints about morale, reinforcing their original belief. Yet the same complaints might also be expected if unclear priorities, inadequate software, unrealistic deadlines or poor coordination were the real causes.
Nothing in the evidence separates these explanations.
This is the central danger of positive test strategy: finding what your theory expects is not the same as showing your theory is uniquely supported. [UC San Diego Pages]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
Peter Wason’s famous “2-4-6” rule-discovery experiment illustrates the problem. Participants were shown the sequence 2-4-6 and asked to discover the hidden rule by proposing other number triples. Most guessed a narrow rule such as “numbers increasing by two” and then tested similar examples like 8-10-12. Because these examples satisfied both their guessed rule and the actual rule (“any ascending numbers”), they kept receiving confirmation without discovering their mistake. Very few proposed deliberately contrasting cases such as 3-5-7 or 8-7-9, which would have revealed that their initial explanation was unnecessarily specific. [Wikipedia+2Wikipedia]WikipediaConfirmation biasConfirmation bias
The evidence kept saying “yes”, but it was saying “yes” to several explanations at once.
Why confirmation feels more convincing than it is
Positive testing creates an illusion of strong evidence because each successful prediction feels like another independent success. In reality, repeated confirmations may add little information if every plausible explanation predicts the same outcome.
This happens because people often ask questions such as:
- “Does this fit my explanation?”
- “Can I find another example?”
- “Is there more evidence of the pattern I already expect?”
These questions rarely reveal whether competing explanations would have succeeded equally well.
Raymond Nickerson’s review of confirmation bias emphasises that people often acquire and interpret evidence selectively, without deliberately intending to be biased. The issue is not simply wanting to be right; it is that attention naturally follows the currently active explanation, making alternative searches less likely. [UC San Diego Pages]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
The result is overconfidence built from evidence that is compatible rather than discriminating.
The hidden question: would another explanation predict this too?
A useful way to evaluate any confirming observation is to ask a second question immediately afterwards:
“If my main alternative were true instead, would I expect to see exactly the same evidence?”
If the answer is yes, the observation provides little help in choosing between explanations.
For example:
ObservationFavourite explanationPlausible rivalDoes the observation discriminate?Customer satisfaction fellProduct quality declinedExpectations increasedNo—both predict complaintsTeam missed deadlinesTeam is lazyProject scope kept changingNo—both predict delaysWebsite traffic increasedAdvertising workedSeasonal demand roseNo—both predict higher visits
The observation becomes valuable only when it is substantially more likely under one explanation than another.
This shifts attention from collecting supporting examples to comparing predictive power.
How to add rival-focused checks without overcorrecting
Avoiding the trap does not require abandoning positive testing altogether. Instead, pair it with deliberately chosen evidence that competing explanations predict differently.
Three practical habits are especially effective.
Ask one rival question for every favourite explanation.
Instead of searching only for evidence that supports your idea, ask what another reasonable explanation would expect you to find.
Look for separating evidence.
Prioritise observations that would differ depending on which explanation is correct. If every candidate predicts the same result, keep searching until you identify evidence that genuinely distinguishes them.
Test boundary cases.
Investigate situations where your explanation predicts success but a rival predicts failure, or vice versa. These cases carry far more information than collecting additional ordinary examples.
This is not the same as searching obsessively for disconfirmation. Klayman and Ha argued that positive testing often remains efficient; the improvement comes from choosing diagnostic tests rather than merely confirming ones. [Zurich Psych Institute]psychologie.uzh.chZurich Psych InstituteHypothesis Generation, Sparse Categories, and the…by DJ Navarro · Cited by 113 — In general, people prefer to em…
A practical comparison
The difference between weak and strong testing can be summarised simply.
Weak positive searchStrong comparative searchLooks where the favourite explanation predicts evidenceLooks where competing explanations predict different evidenceCounts confirming examplesCompares which explanation predicts observations betterGains confidence from repeated agreementGains confidence from evidence that separates alternativesStops after finding expected evidenceContinues until rivals become less plausible
The second approach is slower initially, but it produces conclusions that are more resilient when new information appears.
The real goal is informative evidence, not supportive evidence
The positive test strategy survives because it often works well enough. Most everyday reasoning would become painfully slow if every small judgement required exhaustive attempts at refutation. The mistake is not beginning with your favourite explanation. The mistake is never leaving it.
Within the broader practice of keeping live alternatives available, the important shift is from asking whether evidence fits your story to asking whether it fits your story better than credible rivals. Evidence becomes genuinely informative when it changes the balance between explanations, not merely when it agrees with the one you already expected. UC San Diego Pages+2Zurich Psych Institute [pages.ucsd.edu]pages.ucsd.eduUC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in ManyUC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many…October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi…
Amazon book picks
Further Reading
Books and field guides related to The Search That Confirms Too Easily. Use these as the next step if you want deeper reading beyond the article.
The Scout Mindset
Focuses on seeking disconfirming evidence and better hypothesis testing.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
-
Source: Wikipedia
Title: Confirmation bias
Link: https://en.wikipedia.org/wiki/Confirmation_bias -
Source: Wikipedia
Title: Peter Cathcart Wason
Link: https://en.wikipedia.org/wiki/Peter_Cathcart_Wason -
Source: pages.ucsd.edu
Title: UC San Diego Pages Confirmation Bias: A Ubiquitous Phenomenon in Many
Link: https://pages.ucsd.edu/~mckenzie/nickersonConfirmationBias.pdfSource snippet
UC San Diego PagesConfirmation Bias: A Ubiquitous Phenomenon in Many...October 6, 2004 — by RS Nickerson · 1998 · Cited by 12458 — Confi...
Published: October 6, 2004
-
Source: psychologie.uzh.ch
Link: https://www.psychologie.uzh.ch/dam/jcr%3Af969345c-6578-4d10-8a88-f479aa0c0190/Navarro.PsychRev.2011.pdfSource snippet
Zurich Psych InstituteHypothesis Generation, Sparse Categories, and the...by DJ Navarro · Cited by 113 — In general, people prefer to em...
Additional References
-
Source: explorable.com
Link: https://explorable.com/confirmation-biasSource snippet
Confirmation Bias and the Wason Rule Discovery TestWason's Rule Discovery Test proves that most people do not try at all to test their hy...
-
Source: quizlet.com
Link: https://quizlet.com/study-guides/wason-s-2-4-6-task-and-confirmation-bias-insights-88402005-80dc-4ac7-91d9-682d0292f341Source snippet
Wason's 2-4-6 Task and Confirmation Bias Insights15 Mar 2025 — Overview of Wason's Experiment, Understanding Confirmation Bias, Implicati...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11561113/Source snippet
by V Berthet · 2024 · Cited by 26 — When they are asked to test a given hypothesis, individuals tend to be biased towards confirming e...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/380309922_A_common_factor_underlying_confirmation_bias_in_hypothesis_testing_tasksSource snippet
(PDF) A common factor underlying confirmation bias in...2 May 2024 — When they are asked to test a given hypothesis, individuals tend to...
Published: May 2024
-
Source: youtube.com
Title: 3 Ways To Master Analytical Thinking Without Breaking A Sweat
Link: https://www.youtube.com/watch?v=D5Oeajtbg0YSource snippet
3 Ways To Master Analytical Thinking Without Breaking A Sweat - YouTube...
-
Source: youtube.com
Link: https://www.youtube.com/watch?v=vKA4w2O61XoSource snippet
Music: Two and a Half Minutes by Goodenough ([http://goodenoughmusic.bandcamp.com/album/straight-forward-ep](http://goodenoughmusic.bandcamp.com/album/straight-forward-ep))...
-
Source: u-pad.unimc.it
Link: https://u-pad.unimc.it/retrieve/handle/11393/354972/95567738-8d3e-489f-b0d0-de91d9eca77e/s13421-025-01691-3%282%29.pdfSource snippet
Thinking in opposites improves hypothesis testing...by E Branchini · 2025 · Cited by 3 — Adopting a positive hypothesis test strate...
-
Source: youtube.com
Title: The Reasoning Test Psychologists Still Can’t Explain
Link: https://www.youtube.com/watch?v=XOiCFa5niuMSource snippet
ONLY 4% of People Get This CORRECT! - (The Wason Selection Task)...
-
Source: youtube.com
Title: Can you guess the pattern?
Link: https://www.youtube.com/watch?v=4512ALn5YdYSource snippet
3 Ways To Master Analytical Thinking Without Breaking A Sweat...
-
Source: youtube.com
Title: ONLY 4% of People Get This CORRECT!
Link: https://www.youtube.com/watch?v=9wIm2pSlRQcSource snippet
The Confirmation Bias...
Topic Tree