Branchoria Sharper Thinking Sharper Thinking

Practical ways to reason better, test ideas, and make decisions under uncertainty.

Within Sharper Thinking

What Evidence Would Change Your Mind?

Deciding in advance what evidence would matter protects you from searching only for support.

On this page

  • Defining evidence before searching
  • Strong, weak, and irrelevant signals
  • Using thresholds in personal decisions
Preview for What Evidence Would Change Your Mind?

Introduction

Evidence standards are rules you set before you start looking: what would count as a real signal, what would count against your view, and how much would be enough to change your mind. They matter because people do not simply “see the facts” and update cleanly. Research on confirmation bias, biased assimilation and motivated belief updating shows that people often search for supportive information, scrutinise unwelcome findings more harshly, and give more weight to information they want to be true. [PMC+2Frank Baumgartner]pmc.ncbi.nlm.nih.govHumans actively sample evidence to support prior beliefs - PMCby P Kaanders · 2022 · Cited by 68 — Previous research has shown that pa…

Overview image for Evidence Tests The practical antidote is not to become perfectly neutral. It is to make the test visible before your preferences take over. A useful evidence standard says: “If I saw this kind of evidence, from this kind of source, with this level of reliability, I would reduce, raise, or reverse my confidence.” That habit turns thinking from a search for reassurance into a disciplined way of learning.

Decide the test before the search begins

The most important moment in evidence-based thinking often happens before any evidence is collected. Once you have started searching, your attention, wording and judgement are already shaped by what you hope to find. In online search experiments, researchers have found that people’s prior attitudes can influence what information they choose to view and how they move through search results, which makes “I’ll just look it up” less neutral than it sounds. [UCSD Pages]pages.ucsd.eduPages Confirmation Bias: A Ubiquitous Phenomenon in ManyPages Confirmation Bias: A Ubiquitous Phenomenon in Many

A pre-set evidence standard works like a small personal version of preregistration in science. In formal research, preregistration means creating a time-stamped plan for hypotheses, methods and analyses before collecting or analysing data, so that planned tests can be distinguished from later exploration. The Center for Open Science describes this as a way to separate hypothesis generation from hypothesis testing, because the same data can otherwise be used to invent and then “confirm” a claim. [cos.io]cos.ioOpen source on cos.io.

For ordinary thinking, the same principle can be simpler. Before reading reviews, arguing about a policy, judging a work proposal or deciding whether a health claim is credible, write one sentence:

“I currently think X; I would change my mind if Y.”

The value is not that your first threshold will be perfect. It is that you can later catch yourself moving the goalposts. Without a prior standard, almost any outcome can be reinterpreted: a small positive study becomes “promising”, a failed replication becomes “not the right context”, and a critic becomes “biased”. With a prior standard, you have to decide whether the evidence met the test you said would matter.

A good evidence test usually includes four parts:

  1. The claim. Make it specific enough to be tested. “This app is useful” is vague; “This app will save me at least two hours a week after a month of use” is testable.
  2. The evidence type. Decide what would count: controlled trial, independent review, direct measurement, expert consensus, repeated personal experience, audit trail, or some other signal.
  3. The threshold. State how much is enough: one decisive observation, three independent examples, a measurable improvement, a confidence interval that excludes trivial effects, or a pattern that survives alternative explanations.
  4. The action. Say what would change: belief, confidence, next step, budget, vote, purchase, apology, or abandonment of the idea.

This is not a demand for laboratory-grade evidence in every part of life. It is a demand that the strength of the evidence match the size and reversibility of the decision.

Evidence Tests illustration 1

Strong, weak and irrelevant signals are not the same thing

A common failure in everyday reasoning is treating all “evidence” as if it belongs in one pile. A story from a friend, a randomised controlled trial, a repeated personal measurement, a company white paper and a viral screenshot can all feel like evidence, but they do not carry the same weight. The question is not whether something is evidence in the broadest sense. The question is whether it is diagnostic: does it genuinely make the claim more or less likely?

Health evidence frameworks make this distinction explicit. The GRADE approach, used in systematic reviews and guidelines, rates certainty across domains such as risk of bias, inconsistency, indirectness, imprecision and publication bias. Cochrane summarises GRADE certainty as high, moderate, low or very low, depending on how much confidence reviewers can place in the estimated effect. [Cochrane]cochrane.orgChapter 14: CompletingChapter 14: Completing

Those categories translate well beyond medicine. A strong signal is not just “a fact I like”. It is evidence that is relatively hard for the false version of the claim to explain.

Strong signals usually have several of these features:

  • Independence: the evidence does not all come from the same person, incentive system or copied source.
  • Specificity: it predicts or measures the exact thing at issue, not a vague cousin of it.
  • Comparability: it uses a fair comparison group, baseline or counterfactual.
  • Replicability: the pattern appears again under similar conditions.
  • Resistance to easy alternative explanations: bias, chance, selection effects or measurement error are less likely to explain the result.
  • Costliness: the source would pay a real reputational, financial or practical cost if the claim were false.

Weak signals may still be useful, especially early on, but should not do heavy work. A testimonial can suggest what to investigate next, but it is a poor basis for concluding that a treatment works. A single month of improved sales may be encouraging, but it may reflect seasonality, discounting or a large customer that will not repeat. A confident expert opinion may be worth hearing, but it is weaker when the expert has not seen the relevant data or when other qualified experts disagree.

Irrelevant signals are the most deceptive because they often feel persuasive. A famous person endorsing an idea, a beautiful chart without clear sourcing, a large number of likes, or a technically true fact that does not address the claim may all create confidence without adding much diagnostic value. The standard question is: “Would I expect to see this signal even if the claim were false?” If the answer is yes, the signal should move your confidence little or not at all.

OSF and Preregistration

Channel: Dominic Lemken

Open on YouTube

Why mixed evidence often hardens the view you already had

One reason to set evidence standards in advance is that mixed evidence is unusually easy to misuse. In a classic 1979 study, Charles Lord, Lee Ross and Mark Lepper showed people with opposing views on capital punishment mixed evidence about whether it deterred murder. Participants tended to rate studies supporting their prior view as more convincing and to criticise opposing studies more sharply, even though both sides were seeing the same broad evidential situation. [Frank Baumgartner]fbaum.unc.edujpsp 1979 Lord Ross LepperROSS, AND M. LEPPER attitudes and beliefs did change after each new piece of evidence was presented, and from the beginning to the end of…

This does not mean people are hopelessly irrational. Sometimes scepticism towards disconfirming evidence is justified: a weak study should not overturn a well-supported belief. The problem is asymmetry. If supportive evidence gets a light inspection and opposing evidence gets a forensic audit, your “standard” is not an evidence standard at all. It is a defence system.

A useful test is to reverse the result:

“If this same method had produced the opposite conclusion, would I treat it as persuasive?”

If you would dismiss the method when it hurts your side, you should not rely on it heavily when it helps. This is especially important for emotionally charged subjects, personal identity claims, workplace conflicts and decisions where you have already argued publicly for one option. Public commitment can make updating harder because changing your mind now also feels like losing status.

Actively open-minded thinking research points to the same discipline: good thinking involves seeking and processing information that could disconfirm one’s beliefs, not merely collecting arguments for the preferred answer. Stanovich and colleagues describe actively open-minded thinking as a disposition involving willingness to consider alternatives, sensitivity to evidence, and readiness to revise beliefs. [PMC]pmc.ncbi.nlm.nih.govPMCActively Open-Minded Thinking and Its MeasurementPMCActively Open-Minded Thinking and Its Measurement

Evidence Tests illustration 2

Use thresholds, not vibes, for personal decisions

Evidence standards become most useful when they are tied to a decision threshold. A threshold is the point at which evidence is strong enough to justify an action. The threshold should change with stakes. You do not need the same standard to try a new breakfast as to change medication, accuse someone of dishonesty, invest savings, leave a job or publish a serious allegation.

NICE, the UK body that produces health and care guidance, frames good decisions as combining evidence with a person’s values, preferences and circumstances. That matters because evidence rarely decides everything by itself. It tells you what is likely, how certain the estimate is, and what trade-offs are involved; the decision threshold also depends on cost, risk, reversibility and what matters to the person affected. [NICE]nice.org.ukOpen source on nice.org.uk.

For everyday use, three threshold levels are often enough:

Low threshold: reversible, low-cost experiments.

Use this when the downside is small and the decision is easy to undo. Trying a new note-taking method, rearranging your morning routine, or testing a different way to run a weekly meeting does not need overwhelming proof. A plausible mechanism, a small trial and a clear review date may be enough.

Medium threshold: meaningful cost or opportunity cost.

Use this when the decision consumes money, time, reputation or attention. Before buying an expensive course, changing a team process or committing to a side project, look for independent evidence, comparable examples, realistic failure rates and signs that the benefits apply to your situation.

High threshold: serious, irreversible or harm-bearing decisions.

Use this for medical, legal, financial, safety or relationship decisions where a false belief could cause major damage. Here, anecdotes and confidence are not enough. You need higher-quality evidence, professional input where appropriate, explicit uncertainty, and a serious attempt to find disconfirming information.

The threshold should also be lower for changing confidence than for taking action. You can reduce confidence in a belief after moderate evidence without immediately doing anything dramatic. For example, one credible failed replication may not prove a claim false, but it should weaken your confidence and make you more cautious about relying on it.

Julia Galef: Think Rationally via Bayes' Rule | Big Think

Channel: Big Think

Open on YouTube

A practical evidence test for everyday reasoning

The simplest evidence standard is a short table you fill in before searching. It is especially useful when you feel strongly, when the answer affects money or relationships, or when you have already started defending a conclusion.

QuestionWhat to write before searchingExampleWhat do I currently believe?State the claim plainly.“This supplier is unreliable.”What would change my mind?Name the evidence that would weaken or reverse the claim.“If delivery records show fewer than 3 late orders in the past 12 months.”What would strengthen my view?Name the evidence that would increase confidence.“If late deliveries are above 15%, confirmed by our logs and customer complaints.”What evidence should not move me much?Identify weak or irrelevant signals.“One annoyed colleague’s memory, unless backed by records.”What threshold fits the decision?Match the standard to the stakes.“Medium: enough to decide whether to renew the contract, not enough for a public accusation.”

This small act changes the search. Instead of asking “Can I find support?”, you ask “Which possible findings would discriminate between my view and its alternatives?” That is the key difference between motivated search and useful analysis.

A good evidence test also protects you from overcorrecting. Open-mindedness is not the same as letting every new claim move you equally. If a claim has already survived strong tests, weak contrary evidence should only move it slightly. If a claim rests mostly on habit, hearsay or one convenient example, even moderate contrary evidence should matter.

Preregistration: Improve Research Rigor, Reduce Bias

Channel: Center for Open Science

Open on YouTube

What to do when evidence is inconclusive

Sometimes the honest answer is that the evidence does not meet any strong threshold. This is not a failure of thinking. It is often the most useful conclusion. Research methods guidance makes a similar distinction between absence of evidence and evidence of absence: a non-significant or unclear result may simply be too imprecise to decide whether an effect is real, trivial or absent. [arXiv]arxiv.orgOpen source on arxiv.org.

In personal decisions, inconclusive evidence should usually lead to one of four responses:

Delay the decision when the cost of waiting is low and better evidence is likely soon.

Run a smaller test when a full commitment is risky but a pilot would produce useful feedback.

Choose the safer default when stakes are high and uncertainty itself is a reason for caution.

Decide anyway, but label the uncertainty when action is necessary despite imperfect evidence.

The worst response is to pretend inconclusive evidence is decisive because a decision feels urgent. The second worst is to keep gathering low-quality evidence long after it has stopped being informative. A threshold helps here too: if you know what quality of evidence would matter, you can stop collecting signals that were never going to change the conclusion.

Evidence Tests illustration 3

The standard should apply to your own side first

The deepest benefit of evidence standards is not that they help you defeat bad arguments from other people. It is that they make your own reasoning harder to fool. Confirmation bias is not limited to gullible people, political opponents or those who lack education. It is a general feature of human information processing, and it becomes sharper when identity, desire or prior commitment is involved. [PMC]pmc.ncbi.nlm.nih.govHumans actively sample evidence to support prior beliefs - PMCby P Kaanders · 2022 · Cited by 68 — Previous research has shown that pa…

A strong personal rule is:

Do not accept evidence for your preferred view that you would reject if it supported the opposite view.

This rule is demanding, but it is also clarifying. It does not require you to give every side equal weight. It requires you to give the same method equal scrutiny. If a tiny sample, anonymous source, vague graph or cherry-picked example would not persuade you against your view, it should not persuade you for it.

That is why “What evidence would change your mind?” is such a powerful question. It reveals whether a belief is connected to reality by a working hinge. If no imaginable evidence would matter, the belief may still be a value, preference, identity commitment or practical stance, but it is no longer being held as an ordinary factual claim. For improving analytical thinking, that distinction is crucial: some commitments are chosen, but factual confidence has to remain answerable to evidence.

Amazon book picks

Further Reading

Books and field guides related to What Evidence Would Change Your Mind?. Use these as the next step if you want deeper reading beyond the article.

Browse more on Amazon: The Scout Mindset Superforecasting Thinking, Fast and Slow

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Live-tested eBay searches with available results related to this page.

Using USA
Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

  1. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC9038198/
    Source snippet

    Humans actively sample evidence to support prior beliefs - PMCby P Kaanders · 2022 · Cited by 68 — Previous research has shown that pa...

  2. Source: pages.ucsd.edu
    Title: Pages Confirmation Bias: A Ubiquitous Phenomenon in Many
    Link: https://pages.ucsd.edu/~mckenzie/nickersonConfirmationBias.pdf

  3. Source: cos.io
    Link: https://www.cos.io/initiatives/prereg

  4. Source: cochrane.org
    Title: Chapter 14: Completing ‘
    Link: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-14

  5. Source: pmc.ncbi.nlm.nih.gov
    Title: PMCActively Open-Minded Thinking and Its Measurement
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC9966223/

  6. Source: nice.org.uk
    Link: https://www.nice.org.uk/what-nice-does/our-guidance/about-nice-guidelines/about-shared-decision-making

  7. Source: nice.org.uk
    Link: https://www.nice.org.uk/what-nice-does/our-guidance/about-nice-guidelines/making-decisions-using-nice-guidelines

  8. Source: arxiv.org
    Link: https://arxiv.org/abs/2305.04587

  9. Source: cochrane.org
    Link: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-15

  10. Source: cos.io
    Title: choosing preregistration template guide for researchers
    Link: https://www.cos.io/blog/choosing-preregistration-template-guide-for-researchers

  11. Source: cos.io
    Link: https://www.cos.io/initiatives/registered-reports

  12. Source: colorectal.cochrane.org
    Title: how to grade
    Link: https://colorectal.cochrane.org/sites/colorectal.cochrane.org/files/uploads/how_to_grade.pdf

  13. Source: netherlands.cochrane.org
    Title: 2 Kwaliteit van de evidence
    Link: https://netherlands.cochrane.org/sites/netherlands.cochrane.org/files/uploads/GRADE_workshop/2_Kwaliteit%20van%20de%20evidence.pdf

  14. Source: arxiv.org
    Link: https://arxiv.org/abs/2206.02266

  15. Source: fbaum.unc.edu
    Title: jpsp 1979 Lord Ross Lepper
    Link: https://fbaum.unc.edu/teaching/articles/jpsp-1979-Lord-Ross-Lepper.pdf
    Source snippet

    ROSS, AND M. LEPPER attitudes and beliefs did change after each new piece of evidence was presented, and from the beginning to the end of...

  16. Source: frontiersin.org
    Link: https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2021.771948/full

  17. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8543928/

  18. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11404843/

  19. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC11495861/

  20. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC3871726/

  21. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7725296/

  22. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC7347922/

  23. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC12618241/

  24. Source: pmc.ncbi.nlm.nih.gov
    Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC8514123/

  25. Source: pubmed.ncbi.nlm.nih.gov
    Link: https://pubmed.ncbi.nlm.nih.gov/33461905/

  26. Source: Wikipedia
    Title: Confirmation bias
    Link: https://en.wikipedia.org/wiki/Confirmation_bias

  27. Source: bndu.ox.ac.uk
    Link: https://www.bndu.ox.ac.uk/sites/default/files/confidence.pdf

  28. Source: nhmrc.gov.au
    Title: assessing certainty evidence
    Link: https://www.nhmrc.gov.au/guidelinesforguidelines/develop/assessing-certainty-evidence

  29. Source: nhmrc.gov.au
    Link: https://www.nhmrc.gov.au/file/23340/download?token=WCPzDa7m

  30. Source: nice.org.uk
    Title: content and process standards for patient decision aids
    Link: https://www.nice.org.uk/corporate/ecd8/chapter/content-and-process-standards-for-patient-decision-aids

  31. Source: frontiersin.org
    Link: https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2025.1565928/full

  32. Source: bvanudgeconsulting.com
    Title: confirmation bias
    Link: https://www.bvanudgeconsulting.com/bias-of-the-week/confirmation-bias/

Additional References

  1. Source: nature.com
    Link: https://www.nature.com/collections/cjjiifhaff

  2. Source: youtube.com
    Title: Julia Galef: Think Rational via Bayes’ Rule | Big Think
    Link: https://www.youtube.com/watch?v=NEqHML98RgU
    Source snippet

    Preregistration: Improve Research Rigor, Reduce Bias...

  3. Source: researchgate.net
    Link: https://www.researchgate.net/publication/339074064%27Consider_the_Opposite%27-Effects_of_Elaborative_Feedback_and_Correct_Answer_Feedback_on_Reducing_Confirmation_Bias-_a_Pre-registered_Study

  4. Source: experimentology.io
    Link: https://experimentology.io/011-prereg.html

  5. Source: sjdm.org
    Link: https://sjdm.org/~jbaron/ms/talks/aot.pdf

  6. Source: researchgate.net
    Link: https://www.researchgate.net/publication/285693848_Actively_open-minded_thinking

  7. Source: semanticscholar.org
    Link: https://www.semanticscholar.org/paper/Biased-Assimilation-and-Attitude-Polarization%3A-The-Lord-Ross/16ae4cf82e87451492d0eb12190acfb63294e305

  8. Source: academia.edu
    Link: https://www.academia.edu/80715337/Biased_assimilation_and_attitude_polarization_The_effects_of_prior_theories_on_subsequently_considered_evidence

  9. Source: fs.blog
    Link: https://fs.blog/confirmation-bias/

  10. Source: thedecisionlab.com
    Link: https://thedecisionlab.com/biases/confirmation-bias

Topic Tree

Follow this branch

Parent topic

Sharper Thinking

Related pages 29

More on this topic 6