Within Claims
Did the Policy Fail, or Just Come First?
A bad outcome after a policy change is not enough to show the policy failed unless competing causes are ruled out.
On this page
- Why timing feels persuasive
- Coincidence, background trends, and rival causes
- Evidence that strengthens a policy failure claim
Page outline Jump by section
Introduction
When a policy is introduced and a bad outcome follows, many people instinctively connect the two. A rise in crime after a policing reform, higher prices after a regulation, or worse exam results after an education change can all seem like straightforward evidence that the policy failed. The timing feels convincing because the policy came first and the problem came second.
Yet timing alone is rarely enough to establish cause and effect. In everyday policy debates, one of the most common reasoning errors is to treat a before-and-after sequence as proof that the policy produced the outcome. To judge whether a policy really failed, we must ask a more demanding question: what would likely have happened if the policy had not been introduced? Answering that question requires considering background trends, rival explanations, and the mechanism by which the policy is supposed to affect outcomes. Modern policy evaluation methods are built around precisely this challenge.[What Works Growth]whatworksgrowth.orgusing impact evaluation evidence in policymakingWhat Works GrowthUsing impact evaluation evidence in policymaking30 Jan 2025 — Impact evaluation evidence helps us understand which polic…
Did the Policy Fail, or Just Come First?
Before-and-after arguments have a simple structure:
- A policy was introduced.
- An outcome occurred afterwards.
- Therefore, the policy caused the outcome.
The problem is that many things change over time. Economies expand and contract, populations age, technologies spread, weather varies, and social behaviour shifts. Any of these factors may influence the outcome being discussed.
Imagine a city introduces a new transport policy in January and traffic congestion worsens by June. The worsening congestion could be caused by the policy. But it could also reflect population growth, major roadworks, changes in commuting patterns, or a broader regional trend. The fact that congestion increased after the policy does not by itself distinguish among these possibilities.
Critical thinking therefore requires separating temporal order from causal evidence. A cause must come before its effect, but not everything that comes before an event is its cause.
Why Timing Feels Persuasive
Humans are naturally sensitive to sequences of events. Detecting patterns is often useful, and in daily life many genuine causes do precede their effects. If a person touches a hot stove and immediately feels pain, timing is an important clue.
Policy interventions, however, occur in complex systems involving thousands or millions of people. Multiple influences operate simultaneously, and effects may emerge gradually rather than instantly. This complexity makes simple before-and-after reasoning much less reliable.
Timing becomes especially persuasive when:
- The outcome is emotionally charged.
- The policy was controversial before implementation.
- The outcome matches prior expectations.
- The policy change was highly visible while other influences were less obvious.
In public debate, a dramatic event shortly after a policy change often receives disproportionate attention. A single negative outcome may be presented as proof of failure even when longer-term data point in a different direction.
This tendency illustrates a broader analytical lesson: evidence that fits a preferred story often feels stronger than it really is.
Coincidence, Background Trends, and Rival Causes
The strongest challenge to a simple before-and-after claim is the existence of alternative explanations.
Background trends can mislead
Suppose unemployment rises after a government introduces a job-training programme. Critics may claim the programme failed. Yet if unemployment was already rising before the programme began, or if a national recession affected many regions simultaneously, the increase may tell us little about the programme itself.
Policy analysts therefore examine trends before the intervention. If the outcome was already moving in the same direction, attributing the change solely to the new policy becomes much harder.[OUP Academic]academic.oup.comOUP AcademicHow to do (or not to do) … Assessing the impact of a policy…by M Lagarde · 2012 · Cited by 386 — The objective of this art…
Other events may occur at the same time
Policies rarely operate in isolation. A tax change may coincide with shifts in consumer confidence. A public-health policy may overlap with seasonal disease patterns. An education reform may occur alongside curriculum changes, teacher shortages, or demographic shifts.
These competing influences are often called confounders or rival causes because they can create the appearance of a policy effect even when the policy itself played little or no role. Researchers studying policy impacts devote substantial effort to identifying and controlling for such factors.[evidenceforaction.org+2PMC]evidenceforaction.orgA Brief Discussion on Time-Varying Confounding in…1 Feb 2021 — Confounding is a major threat to the validity of all observational stud…
Some outcomes fluctuate naturally
Many social indicators rise and fall over time even without intervention. Crime rates, inflation, employment, and health outcomes often show natural variation.
If observers focus only on two points in time—before the policy and after the policy—they may mistake ordinary fluctuation for evidence of success or failure. Looking at longer trends helps reveal whether an apparent change is unusual or simply part of an ongoing pattern.
Evidence That Strengthens a Policy Failure Claim
A before-and-after comparison is not useless. It can provide an important clue. The question is what additional evidence makes the claim more convincing.
A plausible mechanism
A stronger argument explains how the policy could realistically produce the outcome.
For example, if a regulation increases compliance costs for businesses, and those costs are expected to raise prices, then subsequent price increases fit a clear causal pathway. The presence of a believable mechanism does not prove the claim, but it makes it more credible than a mere coincidence in timing.
Comparison with a credible counterfactual
The central challenge in policy evaluation is determining what would have happened without the policy. Because we cannot observe both realities at once, analysts seek comparison groups or alternative benchmarks.
One widely used approach compares changes in an affected group with changes in a similar unaffected group. If outcomes worsen only where the policy was implemented while remaining stable elsewhere, the case for causation becomes stronger. This family of methods, often called difference-in-differences analysis, is designed specifically to move beyond simple before-and-after comparisons.[PMC+2Lippincott Journals]pmc.ncbi.nlm.nih.govAdvances in Difference-in-differences Methods for Policy…by G Wang · 2024 · Cited by 121 — Difference-in-differences (DiD) is a pow…
Multiple observations over time
A stronger assessment uses many observations before and after implementation rather than a single snapshot.
If a policy is followed by a clear and sustained break from a long-established trend, that pattern provides more persuasive evidence than one isolated post-policy data point. Researchers often use interrupted time-series designs to examine whether a trend changed noticeably after a policy was introduced.[OUP Academic]academic.oup.comOUP AcademicHow to do (or not to do) … Assessing the impact of a policy…by M Lagarde · 2012 · Cited by 386 — The objective of this art…
Ruling out rival explanations
Confidence increases when alternative causes are investigated and found insufficient.
Suppose a policy is blamed for rising prices. Analysts would ask:
- Did similar price increases occur in places without the policy?
- Were there supply shocks affecting the entire market?
- Did input costs change at the same time?
- Did broader economic conditions explain the increase?
The more thoroughly competing explanations are examined, the stronger a causal claim becomes.[PMC+2RAND]pmc.ncbi.nlm.nih.govIdentifying Optimal Methods for Addressing Confounding Bias…by BA Griffin · 2023 · Cited by 9 — We conducted a simulation study to…
A Practical Test for News and Public Debate
When encountering a claim that a policy succeeded or failed because conditions changed afterwards, ask four questions:
- What exactly changed after the policy?
- Was the same trend already present beforehand?
- What other events or influences occurred during the same period?
- What evidence shows that a similar group without the policy experienced a different outcome?
If these questions remain unanswered, the argument may rely mainly on timing rather than on convincing causal evidence.
This does not mean the policy worked or failed. It means the available evidence has not yet established which explanation is most likely.
What This Teaches About Everyday Reasoning
Before-and-after evidence is often the starting point of policy evaluation, not the endpoint. A bad outcome following a policy change may signal a genuine problem, but it may also reflect background trends, coincidence, or other causes operating at the same time.
For anyone seeking to improve analytical thinking, the key lesson is simple: when someone says a policy failed because things got worse afterwards, do not stop at the timeline. Ask what evidence connects the policy to the outcome, what rival explanations have been considered, and whether the observed change differs from what would probably have happened anyway. That shift—from noticing sequence to testing causation—is one of the most valuable habits in evaluating public arguments.
Amazon book picks
Further Reading
Books and field guides related to Did the Policy Fail, or Just Come First?. Use these as the next step if you want deeper reading beyond the article.
The Book of Why
Directly addresses the mistake of inferring causation from timing alone and introduces counterfactual thinking central to policy evaluation.
Thinking, Fast and Slow
Explains cognitive biases that make before-and-after policy arguments feel persuasive even when evidence is weak.
The Signal and the Noise
Helps readers think critically about evidence, uncertainty, and separating meaningful effects from coincidental patterns.
Causal Inference
Focuses on methods for distinguishing true policy effects from background trends, confounding factors, and rival explanations.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
1.
Source: academic.oup.com
Link:https://academic.oup.com/heapol/article/27/1/76/659122
Source snippet
OUP AcademicHow to do (or not to do) … Assessing the impact of a policy...by M Lagarde · 2012 · Cited by 386 — The objective of this art...
2.
Source: evidenceforaction.org
Link:https://evidenceforaction.org/blog-posts/brief-discussion-time-varying-confounding-evaluation-programs-practices-or-policies
Source snippet
A Brief Discussion on Time-Varying Confounding in...1 Feb 2021 — Confounding is a major threat to the validity of all observational stud...
3.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC10538408/
Source snippet
Identifying Optimal Methods for Addressing Confounding Bias...by BA Griffin · 2023 · Cited by 9 — We conducted a simulation study to...
4.
Source: rand.org
Link:https://www.rand.org/pubs/external_publications/EP70435.html
Source snippet
Identifying Optimal Methods for Addressing Confounding...17 Apr 2024 — We conducted a simulation study to examine how differing magnitud...
5.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC11305929/
Source snippet
Advances in Difference-in-differences Methods for Policy...by G Wang · 2024 · Cited by 121 — Difference-in-differences (DiD) is a pow...
6.
Source: whatworksgrowth.org
Title: using impact evaluation evidence in policymaking
Link:https://whatworksgrowth.org/resource-library/using-impact-evaluation-evidence-in-policymaking/
Source snippet
What Works GrowthUsing impact evaluation evidence in policymaking30 Jan 2025 — Impact evaluation evidence helps us understand which polic...
7.
Source: journals.lww.com
Title: advances in difference in differences methods for.6.aspx
Link:https://journals.lww.com/epidem/fulltext/2024/09000/advances_in_difference_in_differences_methods_for.6.aspx
Source snippet
Lippincott JournalsAdvances in Difference-in-differences Methods for Policy...by G Wang · 2024 · Cited by 121 — To reduce confounding and...
8.
Source: gov.scot
Title: www.gov.scot Evaluation for policy makers
Link:https://www.gov.scot/publications/evaluation-policy-makers-straightforward-guide/pages/6/
Source snippet
for policy makers - A straightforward guide11 Dec 2018 — Outcome or 'Before and After' evaluation (a data driven approach that uses a bas...
9.
Source: assets.pc.gov.au
Link:https://assets.pc.gov.au/research/supporting/strengthening-evidence/05-chapter4.pdf
Source snippet
This chapter considers what policymakers can do to enable and encourage the production of such evidence.Read more...
10.
Source: projectdelivery.gov.uk
Link:https://projectdelivery.gov.uk/teal-book/home/part-a-project-delivery-in-government/chapter-2-policy-and-evaluation/
Source snippet
Chapter 2. Policy and evaluationCovers the policy process, the three types of evaluation, the roles involved, and what to plan for from t...
Additional References
11.
Source: goltc.org
Link:https://goltc.org/publications/difference-in-differences-approach/
Source snippet
Difference-in-Differences approachThe DiD approach involves the measurement of the change in outcome between two time periods (pre and po...
12.
Source: researchgate.net
Link:https://www.researchgate.net/publication/368968271_Difference-in-Differences_for_Policy_Evaluation
Source snippet
Difference-in-Differences for Policy EvaluationPPM aligned with the DiD design by conceptually distinguishing between pre-and post-enforc...
13.
Source: instituteforgovernment.org.uk
Link:https://www.instituteforgovernment.org.uk/sites/default/files/publications/Policy%20making%20in%20the%20real%20world.pdf
Source snippet
POLICY MAKING IN THE REAL WORLDby P MAKING — It provides both an in-depth look at attempts to reform policy making over the last fourteen...
14.
Source: brookings.edu
Link:https://www.brookings.edu/wp-content/uploads/2016/07/policyandchoice_book.pdf
Source snippet
Policy and choiceTraditional public finance provides a powerful framework for policy analysis, but it relies on a model of human behavior...
15.
Source: ifs.org.uk
Link:https://ifs.org.uk/sites/default/files/output_url_files/oxb08_pol_eval.ppt
Source snippet
oxb08_pol_eval.pptDifference in differences (DD). Suppose a change in policy occurs at time k; We observe agents affected by policy chang...
16.
Source: researchgate.net
Link:https://www.researchgate.net/publication/374101871_Identifying_Optimal_Methods_for_Addressing_Confounding_Bias_When_Estimating_the_Effects_of_State-Level_Policies
Source snippet
ct health-related outcomes are foundational to health and social policy research...
17.
Source: springfieldcentre.com
Link:https://www.springfieldcentre.com/wp-content/uploads/2013/07/Evidence-Based-Policy-and-Systemic-Change1.pdf
Source snippet
and, more acutely, its incompatibility with the latter trend...
18.
Source: youtube.com
Title: An intuitive
Link:https://www.youtube.com/watch?v=J7q2H8aB8bQ
Source snippet
This video is relevant because it explains how a basic "before vs. after" baseline comparison often fails to isolate the true effect of a...
19.
Source: cambridge.org
Link:https://www.cambridge.org/core/journals/national-institute-economic-review/article/state-of-evidencebased-policy-evaluation-and-its-role-in-policy-formation/038B0EC22F4B468D4B2A87C07B11F9B6
Source snippet
The State of Evidence-Based Policy Evaluation and its...by P Davies · 2012 · Cited by 158 — The paper then addresses whether or not evid...
20.
Source: eprints.whiterose.ac.uk
Title: Paper for Policy Studies
Link:https://eprints.whiterose.ac.uk/id/eprint/142911/12/Paper_for_Policy_Studies.pdf
Source snippet
policy success and failure: targets, aims and...by P Begley · 2019 · Cited by 28 — 'Failure' occurs when a policy does not achieve the g...
Topic Tree