Within Cause Check
What Changed Compared With What?
Before-and-after stories become more reliable when they ask what would probably have happened without the change.
On this page
- Why before and after patterns mislead
- Choosing a fair comparison group
- Counterfactual questions in everyday decisions
Page outline Jump by section
Introduction
Before-and-after stories are persuasive because they match how people naturally experience change: something happened, then the outcome improved or worsened. Yet a sequence alone does not show that one event caused the other. The key question is not simply, “Did things change?” but, “What would probably have happened if the change had never been made?” That missing comparison—often called the counterfactual—is the foundation of reliable causal reasoning. [Better Evaluation+2Harvard Data Science Review]betterevaluation.orgBetter EvaluationCompare results to the counterfactual MethodsA statistical model, such as regression analysis, is used to develop an est…
Developing this habit is one of the most practical thinking skills in everyday life. It helps you judge claims about diets, workplace reforms, educational programmes, investments, public policy and personal habits without assuming that every improvement or decline must have been caused by the most visible recent event.
Why before-and-after patterns mislead
A simple before-and-after comparison answers only one question: whether an outcome changed over time. It does not answer why it changed.
Imagine a company introduces a new sales script in March and sales rise in April. The script may deserve credit. Equally, the increase could reflect seasonal demand, an improving economy, a successful advertising campaign or competitors leaving the market. Without a fair comparison, these explanations remain mixed together.
Several common effects make before-and-after evidence especially unreliable.
- Background trends: Outcomes often change anyway. House prices, exam scores, business revenue and disease rates can all move because of broader forces.
- Regression to the mean: Extremely good or bad results often become more typical on the next measurement simply through natural variation. A sports team suffering an unusually poor season may improve the following year even if nothing important changes. Likewise, patients often seek treatment when symptoms are at their worst, making later improvement appear larger than the treatment’s true effect. [Moodle@Units]moodle2.units.itWhen a causal question is of sec- ondary interest to a noncausal question, such as whether services are being.Read more…
- Multiple simultaneous changes: New software, staff turnover, training and organisational restructuring may all occur within weeks of one another.
- Measurement changes: Different definitions, reporting practices or testing methods can create apparent improvements that reflect measurement rather than reality.
These problems do not prove that an intervention failed. They simply mean that the before-and-after comparison alone cannot separate the intervention from everything else happening at the same time.
Choosing a fair comparison group
The strongest comparison asks how similar people, organisations or places performed without the intervention during the same period.
A good comparison group should resemble the treated group in ways that matter for the outcome. The closer the groups are before the intervention, the more believable the comparison afterwards.
In practice, several approaches provide increasingly reliable comparisons.
Randomised comparison
Random assignment gives participants an equal chance of receiving the intervention, helping balance both measured and unmeasured differences across groups. This is why randomised controlled trials are widely regarded as the most reliable design for estimating intervention effects when they are practical and ethical. [Moodle@Units+2Harvard Data Science Review]moodle2.units.itWhen a causal question is of sec- ondary interest to a noncausal question, such as whether services are being.Read more…
Matched comparison
When randomisation is impossible, researchers often compare units that were already similar before the intervention—for example, schools with similar pupil characteristics or patients with comparable medical histories. Matching cannot eliminate every source of bias, but it often produces a fairer comparison than simply comparing everyone who received a treatment with everyone who did not. [PMC]pmc.ncbi.nlm.nih.govMatching methods for causal inference: A review and a look…by EA Stuart · 2010 · Cited by 7781 — Plot of standardized difference of…
Natural comparison over time
Sometimes an intervention affects one region while another similar region does not change. Comparing how both places change over the same period can remove many background trends. This logic underlies difference-in-differences methods, which estimate whether the treated group changed more than would have been expected from broader trends alone. The method depends on important assumptions—particularly that the groups would otherwise have followed similar trends—and should not be treated as automatic proof of causation. [Wikipedia]WikipediaDifference in differencesDifference in differences
The central principle is remarkably simple: compare like with like whenever possible.
Counterfactual questions in everyday decisions
Most people cannot run experiments on their own lives, but they can ask counterfactual questions that make their reasoning much more disciplined.
Instead of saying:
“My productivity improved after I bought a standing desk.”
Ask:
“Would my productivity probably have improved anyway because I started a quieter project, slept better or became more experienced?”
Instead of saying:
“Our marketing campaign caused sales growth.”
Ask:
“How did similar products without the campaign perform during the same period?”
Instead of saying:
“This study app made my child improve.”
Ask:
“How did pupils with similar starting points improve without using the app?”
These questions do not guarantee correct answers. They force you to search for the comparison that your first impression omitted.
Practical ways to build fair comparisons
Outside formal research, perfect comparison groups are rarely available. Even so, several habits substantially improve judgement.
- Compare changes over the same time period rather than different years with different conditions.
- Look for similar people, organisations or places that did not receive the intervention.
- Examine several observations instead of relying on a single dramatic case.
- Check whether the trend had already begun before the intervention.
- Be cautious when the intervention follows an unusually high or unusually low measurement, since regression to the mean may explain part of the change.
- Treat unusually large claimed effects with extra scrutiny unless supported by strong comparative evidence.
The goal is not perfection but reducing obvious sources of mistaken attribution.
Recognising weak comparisons
Certain phrases should immediately prompt scepticism because they rely on incomplete comparisons.
- “Sales increased after we changed the logo.”
- “Crime fell after the new policy.”
- “Everyone felt better after taking the supplement.”
- “The new manager transformed the department.”
Each statement reports a sequence but leaves the comparison unstated. Important follow-up questions include:
- Compared with what happened elsewhere?
- Compared with previous trends?
- Compared with similar people who did not receive the change?
- Compared with what experts expected would have happened anyway?
Only after these comparisons are considered does a causal claim become substantially more credible.
The thinking habit that matters
Fair comparisons are less about statistics than about disciplined reasoning. Every before-and-after claim contains an invisible missing question: what would have happened without the change?
You can never observe both realities for the same person or organisation at the same moment, so causal reasoning depends on finding the best available substitute. Whether through randomised experiments, carefully chosen comparison groups or thoughtful everyday questioning, the quality of a causal conclusion depends far more on the fairness of the comparison than on how dramatic the before-and-after story appears. [Better Evaluation+2Department of Statistics]betterevaluation.orgBetter EvaluationCompare results to the counterfactual MethodsA statistical model, such as regression analysis, is used to develop an est…
Amazon book picks
Further Reading
Books and field guides related to What Changed Compared With What?. Use these as the next step if you want deeper reading beyond the article.
The Book of Why
Directly explains causation, counterfactuals, and why simple before-and-after evidence is insufficient.
Thinking, Fast and Slow
Explores the cognitive biases that make people overinterpret before-and-after patterns and causal stories.
Calling Bullshit
Helps readers spot weak causal claims, misleading comparisons, and flawed reasoning.
Causal Inference
Covers comparison groups, counterfactuals, natural experiments, and practical causal inference.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: moodle2.units.it
Link: https://moodle2.units.it/pluginfile.php/132646/mod_resource/content/1/Estratto_ShadishCookCampbellExperimental2002.pdfSource snippet
When a causal question is of sec- ondary interest to a noncausal question, such as whether services are being.Read more...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC2943670/Source snippet
Matching methods for causal inference: A review and a look...by EA Stuart · 2010 · Cited by 7781 — Plot of standardized difference of...
-
Source: Wikipedia
Title: Difference in differences
Link: https://en.wikipedia.org/wiki/Difference_in_differences -
Source: youtube.com
Title: An intuitive
Link: https://www.youtube.com/watch?v=J7q2H8aB8bQSource snippet
Difference-in-differences methods...
-
Source: youtube.com
Title: Difference-in-differences methods
Link: https://www.youtube.com/watch?v=w56HI8YxLMQSource snippet
Effect Size, Causality, and Research Design...
-
Source: betterevaluation.org
Link: https://www.betterevaluation.org/generate/framework/471/pdfSource snippet
Better EvaluationCompare results to the counterfactual MethodsA statistical model, such as regression analysis, is used to develop an est...
-
Source: hdsr.mitpress.mit.edu
Link: https://hdsr.mitpress.mit.edu/pub/1ybwbmlwSource snippet
Harvard Data Science ReviewCausation, Comparison, and Regressionby A Chattopadhyay · 2024 · Cited by 8 — Comparison and contrast are the...
-
Source: stat.columbia.edu
Link: https://www.stat.columbia.edu/~gelman/arm/chap9.pdfSource snippet
Department of StatisticsCausal inference using regression on the treatment variableTo motivate the detailed study of regression models fo...
Additional References
-
Source: marginaleffects.com
Link: https://marginaleffects.com/chapters/comparisons.htmlSource snippet
6 Counterfactual comparisons – Model to MeaningA counterfactual comparison is a function of two or more model-based [predictions]({{ 'predictions/' | relative_url }}), made wit...
-
Source: arxiv.org
Link: https://arxiv.org/html/2505.13770v2Source snippet
Benchmarking LLMs Against Statistical Pitfalls in Causal...4 Mar 2026 — Reliable causal inference is essential for making decisions in h...
-
Source: arxiv.org
Link: https://arxiv.org/abs/1901.01869 -
Source: stats.stackexchange.com
Title: Both can’t be
Link: https://stats.stackexchange.com/questions/615029/causal-counterfactual-inference-model-comparisonSource snippet
counterfactual inference model comparison5 May 2023 — When refuting two causal models, model 1 has a bigger p-value and an estimated effe...
Published: May 2023
-
Source: youtube.com
Title: Effect Size, Causality, and Research Design
Link: https://www.youtube.com/watch?v=BD_fbtS9aqISource snippet
Counterfactuals: Causal Inference Bootcamp...
-
Source: youtube.com
Title: Counterfactuals: Causal Inference Bootcamp
Link: https://www.youtube.com/watch?v=9j_HWkrSxzISource snippet
Causality and counterfactuals...
-
Source: youtube.com
Title: Causality and counterfactuals
Link: https://www.youtube.com/watch?v=zkxHWDAefEw
Topic Tree