Within Better Questions
Are You Measuring the Wrong Success?
Clear outcomes prevent satisfaction scores, anecdotes, and easy metrics from standing in for the result that matters.
On this page
- Why better needs a measurable meaning
- Satisfaction versus performance outcomes
- How to choose the main success measure
Page outline Jump by section
Introduction
A good question does not merely ask whether something is “better”. It defines what better means before evidence is collected. If the outcome is left vague, weak answers can appear convincing because they measure an easier, more flattering result than the one that actually matters. A programme that participants enjoy, a search engine that feels helpful, or a treatment that improves a laboratory measurement may all look successful while failing to improve real-world performance.
Choosing the right outcome measure is therefore one of the most effective ways to improve analytical thinking. It prevents evidence from drifting towards convenient proxies, ensures that competing claims are judged against the same standard, and makes it much harder for impressive-looking but irrelevant evidence to win the argument.
Why “better” needs a measurable meaning
Many disputes are not really about the evidence. They are about which outcome counts as success.
Consider the question, “Does this writing course work?” Depending on the chosen outcome, the answer could legitimately be:
- Yes, because learners rated it highly.
- Yes, because test scores increased immediately afterwards.
- No, because workplace writing quality did not improve six months later.
- No, because productivity stayed unchanged despite improved knowledge.
All of these findings could be true simultaneously because they measure different outcomes.
This is why evidence-based disciplines ask researchers to specify a primary outcome before collecting data. In clinical research, for example, studies distinguish between primary outcomes (the main question the trial is designed to answer) and secondary outcomes that provide supporting information. Changing emphasis after seeing the results creates opportunities for selective reporting and exaggerated claims. [U.S. Food and Drug Administration]fda.govFood and Drug AdministrationGuidance for Industry:Patient-Reported Outcome MeasuresDecember 8, 2009 — For example, a clinical trial may i…
The practical lesson extends well beyond science: define the outcome before looking for evidence, not afterwards.
Satisfaction versus performance outcomes
One of the most common mistakes is treating satisfaction as though it proves effectiveness.
A positive experience certainly matters. Happy customers return. Engaged students are more likely to continue learning. Patients deserve respectful care. However, satisfaction measures something different from performance.
Training evaluation illustrates this distinction particularly well. The widely used Kirkpatrick framework separates four increasingly demanding levels:
- Reaction: Did participants like the experience?
- Learning: Did they acquire knowledge or skills?
- Behaviour: Did they actually change what they do?
- Results: Did those behavioural changes improve organisational or practical outcomes? [Kirkpatrick Partners, LLC.+2PMC]kirkpatrickpartners.comKirkpatrick Partners, LLC.The Kirkpatrick ModelThe Kirkpatrick Model, or four levels of training evaluation, consists of Reaction, Learni…
The important insight is that success at one level does not guarantee success at the next.
For example:
- A workshop may receive excellent feedback while producing no lasting behaviour change.
- Employees may pass a knowledge test yet continue using old working habits.
- New procedures may be adopted but fail to improve customer outcomes.
Reaction measures are valuable because they can identify delivery problems or barriers to engagement. They become misleading only when they are presented as evidence that the intervention itself achieved its intended purpose.
When proxy outcomes replace the real outcome
Weak answers often rely on proxy or surrogate outcomes—measurements that are easier or quicker to collect than the result people actually care about.
Sometimes proxies are necessary. Waiting years for the ultimate outcome may be impractical. However, proxies deserve confidence only when there is strong evidence that improving the proxy reliably predicts improvement in the real outcome.
Healthcare provides a clear example.
Researchers sometimes measure blood pressure, cholesterol or tumour shrinkage because waiting for differences in survival or quality of life would take much longer. These surrogate endpoints are useful only when evidence shows that changes in the surrogate consistently predict meaningful patient benefits. Regulators and health technology assessment organisations therefore treat surrogate outcomes cautiously rather than assuming they always represent genuine clinical improvement. [Wikipedia+2Value in Health]WikipediaSurrogate endpointSurrogate endpoint
The same reasoning applies outside medicine.
Suppose an organisation claims that introducing an AI search assistant improved decision quality because staff completed searches more quickly. Faster searching is a legitimate measure, but it is only a proxy. If users now make more errors, overlook contradictory evidence, or become overconfident, the faster searches may represent worse decisions rather than better ones.
The closer an outcome is to the decision-maker’s real objective, the harder it becomes to create misleading success stories.
Measuring what matters to the people affected
Another protection against weak evidence is ensuring that outcomes reflect the perspective of the people experiencing the intervention rather than only the organisation delivering it.
Healthcare increasingly incorporates patient-reported outcome measures (PROMs) because laboratory values alone cannot fully describe whether patients actually feel better, function better or experience improved quality of life. Regulators and researchers increasingly recognise that patient-centred outcomes provide information unavailable from clinical measurements alone. [PMC+2dcricollab.dcri.duke.edu]pmc.ncbi.nlm.nih.govThe importance of patient-reported outcomes in clinical trials…by R Mercieca-Bebber · 2018 · Cited by 759 — Patient-reported outcom…
This principle generalises to many decisions.
A transport project should not be judged only by construction speed if travellers ultimately care about journey reliability.
An education programme should not be judged only by attendance if learners ultimately need durable competence.
A customer service system should not be judged only by shorter call durations if customers still require repeat contacts.
The best primary outcome reflects the benefit that motivated the decision in the first place.
How to choose the main success measure
Before searching for evidence, ask a small set of disciplined questions.
- What final outcome actually matters? [facebook.com]facebook.comn what patients actually experience during and after…
Name the result you ultimately want, not merely something associated with it.
- Who experiences that outcome?
Different stakeholders often value different results. Decide whose perspective is primary.
- Can the outcome be measured objectively?
Vague ideas such as “better quality” become more useful when translated into observable measures.
- Is this a direct outcome or only a proxy?
If using a proxy, ask what evidence links it to the real objective.
- When should success be measured?
Immediate improvements may disappear over time, while delayed benefits may not appear in short evaluations.
Answering these questions before collecting evidence sharply reduces the chance of accepting persuasive but irrelevant findings.
Practical warning signs that the wrong outcome is being used
Several patterns frequently indicate that evidence is stronger in appearance than in substance.
- The measured outcome is easier than the real goal. Counting clicks instead of completed tasks, or attendance instead of learning.
- Only favourable secondary outcomes are highlighted. The main intended outcome receives little attention or is omitted.
- The outcome changes during the discussion. Claims quietly shift from effectiveness to popularity, from quality to speed, or from long-term results to short-term gains.
- The reported improvement lacks practical significance. A statistically detectable change may still be too small to matter in everyday use.
- The measure rewards the metric rather than the purpose. Once people optimise the score itself, the score may stop representing genuine success.
These warning signs do not automatically invalidate evidence, but they should prompt careful examination of whether the reported outcome genuinely answers the original question.
Better outcome measures produce better thinking
Strong analytical thinking depends as much on selecting the right outcome as on evaluating the evidence itself. When the desired result is clearly defined before searching begins, impressive anecdotes, satisfaction surveys and convenient proxy measures lose much of their persuasive power.
The most reliable evidence answers the question that actually matters—not merely the question that happened to be easiest to measure.
Amazon book picks
Further Reading
Books and field guides related to Are You Measuring the Wrong Success?. Use these as the next step if you want deeper reading beyond the article.
How to Measure Anything
Directly addresses defining measurable outcomes instead of relying on vague proxies.
The Tyranny of Metrics
Explains how choosing the wrong metrics can distort decisions and apparent success.
Thinking, Fast and Slow
Provides the cognitive framework for evaluating evidence and avoiding misleading judgments.
The Book of Why
Helps readers distinguish meaningful outcomes from misleading associations and proxies.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: fda.gov
Link: https://www.fda.gov/media/77832/downloadSource snippet
Food and Drug AdministrationGuidance for Industry:Patient-Reported Outcome MeasuresDecember 8, 2009 — For example, a clinical trial may i...
Published: December 8, 2009
-
Source: Wikipedia
Title: Outcome measure
Link: https://en.wikipedia.org/wiki/Outcome_measure -
Source: pmc.ncbi.nlm.nih.gov
Title: This level seeks to determine if an increase in company
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC3070232/Source snippet
Employing Kirkpatrick's Evaluation Framework to Determine...by DN Rouse · 2011 · Cited by 274 — Kirkpatrick's model stresses evaluati...
-
Source: Wikipedia
Title: Surrogate endpoint
Link: https://en.wikipedia.org/wiki/Surrogate_endpoint -
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC6219423/Source snippet
The importance of patient-reported outcomes in clinical trials...by R Mercieca-Bebber · 2018 · Cited by 759 — Patient-reported outcom...
-
Source: dcricollab.dcri.duke.edu
Title: Patient-Reported Outcomes
Link: https://dcricollab.dcri.duke.edu/sites/NIHKR/KR/PRO%20Resource%20Chapter.pdfSource snippet
Introduction. This resource white paper was developed in 2014 to introduce clinical researchers to the definition and use of patient-repo...
-
Source: hspr.cuhk.edu.hk
Title: Concept of Patient Reported Outcomes for Attendees update
Link: https://hspr.cuhk.edu.hk/wp-content/uploads/Concept-of-Patient-Reported-Outcomes_for-Attendees_update.pdfSource snippet
Concept of Patient Reported Outcomes13 Jul 2025 — Clinical outcomes are cornerstone evidence for evaluating therapeutic benefits and risk...
-
Source: kirkpatrickpartners.com
Link: https://www.kirkpatrickpartners.com/the-kirkpatrick-model/Source snippet
Kirkpatrick Partners, LLC.The Kirkpatrick ModelThe Kirkpatrick Model, or four levels of training evaluation, consists of Reaction, Learni...
-
Source: valueinhealthjournal.com
Link: https://www.valueinhealthjournal.com/article/S1098-3015%2826%2900050-1/fulltextSource snippet
Methods for Evaluation of Surrogate Endpoints...by S Bujkiewicz · 2026 · Cited by 1 — This ISPOR Task Force report addresses how surroga...
Additional References
-
Source: klarahr.com
Link: https://klarahr.com/en/resources/blog/the-kirk-patrick-modele-valuating-the-impact-of-your-training-programmes-across-four-levelsSource snippet
The Kirkpatrick Model: Evaluating Your Training Programmes4 Oct 2022 — The Kirkpatrick model is a professional training evaluation framew...
-
Source: employment-studies.co.uk
Link: https://www.employment-studies.co.uk/system/files/resources/files/392.pdfSource snippet
might imply about the process of learning and changing behaviour, and we also review some of the research evidence on training...Read more...
-
Source: educationaltechnology.net
Link: https://educationaltechnology.net/kirkpatrick-model-four-levels-learning-evaluation/ -
Source: ispor.org
Title: task force addresses critical gap in surrogate endpoint guidance
Link: https://www.ispor.org/heor-resources/news-top/news/2026/05/28/ispor-task-force-addresses-critical-gap-in-surrogate-endpoint-guidanceSource snippet
ISPOR Task Force Addresses Critical Gap in Surrogate...28 May 2026 — The report will outline different methods to deal with the use of s...
Published: May 2026
-
Source: europepmc.org
Link: https://europepmc.org/article/pmc/3070232Source snippet
Employing Kirkpatrick's evaluation framework to determine...by DN Rouse · 2011 · Cited by 274 — Kirkpatrick's model stresses evaluation...
-
Source: esmed.org
Title: evaluating medical curriculum kirkpatricks approach
Link: https://esmed.org/evaluating-medical-curriculum-kirkpatricks-approach/Source snippet
Evaluating Medical Curriculum: Kirkpatrick's Approachby M Kusmiati · 2025 · Cited by 1 — It outlines the four levels of evaluation define...
-
Source: wecanadvocate.eu
Link: https://wecanadvocate.eu/academy/patient-reported-outcomes-and-other-patient-relevant-measures-and-endpoints/Source snippet
mes to record the patient's lived experience and burden of a disease.Read more...
-
Source: unife.it
Title: yardley kirkpatricks levels and education evidence
Link: https://www.unife.it/medicina/lm.riabilitazione/studiare/minisiti/pianificazione_gestione_proc-edu/modulo-di-processi-educativi-delladulto/2016-17/yardley_kirkpatricks-levels-and-education-evidence.pdfSource snippet
Kirkpatricks levels and education evidenceby S Yardley · 2011 · Cited by 636 — This study aims to review, criti- cally, the suitability o...
-
Source: facebook.com
Link: https://www.facebook.com/EuropeanCancer/posts/patient-reported-outcome-measures-proms-can-help-cancer-professionals-reflect-on/1461357926004588/Source snippet
n what patients actually experience during and after...
-
Source: youtube.com
Title: Primary vs Secondary outcomes
Link: https://www.youtube.com/watch?v=pXesXYmScgISource snippet
Hard Facts, Dangerous Half-Truths, and Total Nonsense by Jeffrey Pfeffer: 6 Minute Summary...
Topic Tree