Within Evidence Tests
Why Health Claims Need Higher Standards
Health claims need higher standards because personal stakes are high and weak testimonials can feel more persuasive than they are.
On this page
- Why testimonials are weak evidence for treatment effects
- How GRADE style certainty helps compare claims
- Setting safer thresholds before changing health decisions
Page outline Jump by section
Introduction
Health claims deserve a higher evidential standard than most other claims because mistakes can directly affect health, finances and quality of life. A supplement that appears harmless may delay effective treatment. A glowing review may persuade someone to spend years on an ineffective therapy. A dramatic before-and-after photograph may represent an unusual outcome rather than the expected one. Improving analytical thinking in this domain therefore requires resisting persuasive stories until they are supported by reliable evidence.
Evidence-based medicine was developed largely because human intuition performs poorly when judging treatment effects. Symptoms naturally fluctuate, many illnesses improve without intervention, expectations influence perceived improvement, and people who experience dramatic benefits are much more likely to share their stories than those who do not. Modern frameworks such as GRADE (Grading of Recommendations, Assessment, Development and Evaluation), used by organisations including Cochrane, the World Health Organization (WHO) and many national guideline developers, were designed to separate genuine treatment effects from these common sources of error. [Cochrane+2gradeworkinggroup.org]cochrane.orgChapter 14: Completing 'Summary of findings' tables and…The GRADE approach categorizes the certainty in a body of evidence as…
Why testimonials rarely establish whether a treatment works
Personal testimonials answer one question well: “What was this person’s experience?” They answer a very different question poorly: “Did the treatment cause the improvement?”
Several factors make individual experiences unreliable measures of treatment effectiveness.
First, many medical conditions improve naturally. Viral infections resolve, episodes of back pain often lessen over time, migraines fluctuate, and many skin conditions improve without any intervention. When improvement happens after a treatment begins, people naturally attribute recovery to the treatment rather than to the underlying course of the illness.
Second, regression towards the mean creates an illusion of effectiveness. People usually seek treatment when symptoms are unusually severe. Because unusually bad days are statistically likely to be followed by more typical days, improvement frequently occurs regardless of whether the intervention had any effect.
Third, placebo responses and expectation effects influence how symptoms are perceived. Pain, fatigue, anxiety and other subjective outcomes are especially susceptible to changes in expectation, making untreated comparisons essential for determining whether a treatment itself is responsible.
Finally, memory is selective. People often remember successful treatments while forgetting unsuccessful ones, and individuals who benefit are far more likely to post reviews or recommend products than those who experience no improvement.
These problems do not imply that every testimonial is false. Rather, they explain why sincere accounts cannot reliably distinguish causation from coincidence. This is precisely why controlled clinical studies compare treated and untreated groups under similar conditions. [Cochrane]cochrane.orgChapter 14: Completing 'Summary of findings' tables and…The GRADE approach categorizes the certainty in a body of evidence as…
Reviews and before-and-after photographs can be systematically misleading
Online health reviews often create an impression of overwhelming evidence even when they contain almost no scientifically useful information.
Most review platforms suffer from strong self-selection bias. Consumers experiencing unusually positive or unusually negative outcomes are much more likely to leave reviews than those with ordinary experiences. Consequently, average ratings frequently do not represent typical outcomes.
Before-and-after photographs present additional problems.
Photographs can differ because of:
- different lighting
- different camera angles
- altered posture
- facial expressions
- timing of measurement
- cosmetic products or grooming
- weight fluctuations unrelated to the advertised treatment
Even when photographs are genuine, they usually document only successful cases. Readers almost never see the much larger group of people who experienced little or no benefit.
For this reason, regulatory agencies generally do not regard testimonials or consumer experiences as scientific substantiation for objective health claims. The United States Federal Trade Commission (FTC), whose guidance influences advertising practice well beyond the United States, states that advertisers must possess “competent and reliable scientific evidence” before making objective health claims. Testimonials may accompany substantiated claims, but they do not substitute for scientific evidence, and surveys of consumer experiences alone are never sufficient to establish treatment effectiveness. [Federal Trade Commission+2Federal Trade Commission]ftc.govhealth products compliance guidanceFederal Trade CommissionHealth Products Compliance Guidance20 Dec 2022 — 2) Before disseminating an ad, advertisers must have adequate su…
Why systematic reviews are stronger than individual studies
Even high-quality clinical trials can produce misleading results by chance. A small positive study may later fail to replicate. Different populations, methods or outcome measures may produce conflicting findings.
Systematic reviews reduce these problems by collecting all relevant evidence according to predefined methods instead of selecting studies that support a preferred conclusion. Well-conducted systematic reviews evaluate study quality, investigate publication bias, assess consistency across findings and often combine results using meta-analysis.
This matters because isolated positive studies frequently attract disproportionate media attention. When viewed alongside all available evidence, the apparent effect may become substantially smaller or disappear entirely.
Cochrane reviews are particularly influential because they follow detailed methodological standards designed to minimise bias in evidence synthesis. Rather than asking whether any study found a benefit, they ask what the totality of reliable evidence indicates. [Cochrane]cochrane.orgChapter 14: Completing 'Summary of findings' tables and…The GRADE approach categorizes the certainty in a body of evidence as…
How GRADE helps compare the certainty of health evidence
One of the most important advances in evidence-based medicine is recognising that not all evidence deserves equal confidence.
The GRADE framework provides a transparent method for judging how certain we should be that an estimated treatment effect is close to the truth. Instead of simply labelling evidence as “good” or “bad”, GRADE evaluates several independent domains that influence confidence.
These include:
- Risk of bias – Were studies well designed and properly conducted?
- Inconsistency – Do independent studies reach similar conclusions?
- Indirectness – Does the evidence directly address the question of interest?
- Imprecision – Are confidence intervals narrow enough to support meaningful conclusions?
- Publication bias – Is there evidence that negative studies were never published?
Randomised controlled trials usually begin as high-certainty evidence because randomisation reduces important biases, whereas observational studies generally begin with lower certainty because differences between groups may explain observed outcomes. However, certainty can be downgraded substantially if trials have methodological flaws or inconsistent findings. Conversely, exceptionally strong observational evidence may occasionally be upgraded if alternative explanations are unlikely. [Cochrane+2Grade Pro]cochrane.orgChapter 14: Completing 'Summary of findings' tables and…The GRADE approach categorizes the certainty in a body of evidence as…
GRADE ultimately classifies certainty into four categories:
- High certainty: Further research is unlikely to change confidence substantially.
- Moderate certainty: New evidence could meaningfully change conclusions.
- Low certainty: Confidence is limited.
- Very low certainty: The true effect may be substantially different from current estimates.
Importantly, these ratings describe confidence in the evidence—not whether a treatment is effective or ineffective.
High-certainty evidence does not always mean large benefits
Another common misunderstanding is equating strong evidence with impressive treatment effects.
A medication may have high-certainty evidence showing that it reduces symptoms only modestly. Conversely, a supplement may advertise dramatic benefits supported only by very low-certainty evidence.
GRADE therefore separates two distinct questions:
- How confident are we that the estimated effect is real?
- How large and clinically important is that effect?
This distinction prevents readers from confusing certainty with magnitude.
A small but well-established benefit may justify treatment in some circumstances. A spectacular claimed benefit supported only by anecdotal reports usually should not.
Recent GRADE guidance also increasingly emphasises explicit decision thresholds, encouraging reviewers to distinguish between trivial, small, moderate and large effects rather than relying solely on statistical significance. [Cochrane]cochrane.orgthresholds and rating certainty of evidence using gradeDecision thresholds guide judgments about…Read more…
Setting safer thresholds before changing health decisions
The parent principle of setting evidence standards before changing your mind becomes particularly valuable in healthcare.
Instead of asking, “Does this sound convincing?”, ask what evidence would actually justify changing behaviour.
For low-risk lifestyle choices, moderate-quality evidence may be sufficient. For expensive treatments, invasive procedures or therapies replacing established medical care, much stronger evidence should normally be required.
A practical hierarchy might look like this:
- Be sceptical of isolated testimonials, celebrity endorsements and influencer experiences.
- Treat before-and-after photographs as illustrations rather than proof.
- Give greater weight to multiple independent randomised trials than to single studies.
- Prefer systematic reviews over individual papers whenever available.
- Consider the certainty rating—not merely whether results reached statistical significance.
- Look for agreement across independent research groups rather than repeated claims from a single commercial source.
This approach mirrors how clinical guideline developers evaluate evidence before recommending treatments rather than relying on persuasive narratives or isolated findings. [CDC+2gradeworkinggroup.org]cdc.govChapter 6: Systematic Review Overview | ACIP GRADE…22 Apr 2024 — This ACIP GRADE handbook provides guidance to the ACIP workgroups…
Recognising common warning signs in health marketing
Commercial health advertising often exploits weaknesses in intuitive reasoning rather than outright fabricating evidence.
Common warning signs include:
- relying primarily on testimonials while offering little published clinical evidence;
- highlighting relative improvements without reporting absolute benefits;
- quoting laboratory or animal studies as though they demonstrated effectiveness in humans;
- emphasising statistical significance while omitting the size of the benefit;
- selectively citing favourable studies while ignoring systematic reviews;
- using vague phrases such as “clinically proven” without identifying the actual research.
The FTC’s Health Products Compliance Guidance emphasises that objective health claims require adequate scientific substantiation before advertising, not after complaints arise. Claims should be supported by competent and reliable scientific evidence appropriate to the nature of the claim, and disclosures cannot cure fundamentally unsupported representations. [Federal Trade Commission+2Federal Trade Commission]ftc.govhealth products compliance guidanceFederal Trade CommissionHealth Products Compliance Guidance20 Dec 2022 — 2) Before disseminating an ad, advertisers must have adequate su…
Applying higher evidence standards without becoming cynical
Higher standards do not require rejecting every new therapy until decades of evidence accumulate. Rather, they encourage proportionate confidence.
When evidence is preliminary, the appropriate response is often “possibly, but uncertain” rather than immediate acceptance or dismissal. As stronger studies accumulate, confidence should rise or fall accordingly.
This mindset makes health decisions more resilient against persuasive anecdotes, selective online reviews and misleading marketing. Instead of asking whether a claim sounds convincing, the more useful question becomes: How certain should I be that this treatment actually produces the claimed benefit? By separating compelling stories from reliable evidence, frameworks such as systematic review and GRADE provide practical tools for making better health decisions while remaining genuinely open to changing one’s mind when stronger evidence appears. [Cochrane+2gradeworkinggroup.org]cochrane.orgChapter 14: Completing 'Summary of findings' tables and…The GRADE approach categorizes the certainty in a body of evidence as…
Amazon book picks
Further Reading
Books and field guides related to Why Health Claims Need Higher Standards. Use these as the next step if you want deeper reading beyond the article.
Testing Treatments
Directly explains why stronger evidence is needed before accepting health claims.
Bad Science
Covers misleading health claims, weak evidence, statistics, and media reporting.
Know Your Chances
Helps readers judge treatment claims using reliable evidence and meaningful statistics.
How to Read a Paper
Explains how to critically evaluate research supporting health interventions.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: cochrane.org
Link: https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-14Source snippet
Chapter 14: Completing 'Summary of findings' tables and...The GRADE approach categorizes the certainty in a body of evidence as...
-
Source: gradeworkinggroup.org
Link: https://www.gradeworkinggroup.org/Source snippet
GRADE homeThe working group has developed a common, sensible and transparent approach to grading quality (or certainty) of evidence and s...
-
Source: cdc.gov
Link: https://www.cdc.gov/acip-grade-handbook/hcp/chapter-6-systemic-review-overview/index.htmlSource snippet
Chapter 6: Systematic Review Overview | ACIP GRADE...22 Apr 2024 — This ACIP GRADE handbook provides guidance to the ACIP workgroups...
-
Source: ftc.gov
Title: health products compliance guidance
Link: https://www.ftc.gov/business-guidance/resources/health-products-compliance-guidanceSource snippet
Federal Trade CommissionHealth Products Compliance Guidance20 Dec 2022 — 2) Before disseminating an ad, advertisers must have adequate su...
-
Source: ftc.gov
Link: https://www.ftc.gov/business-guidance/advertising-marketing/health-claimsSource snippet
Health Claims | Federal Trade CommissionFTC has compiled a list of seven advertising claims that are likely to be a tip-off to deception...
-
Source: cochrane.org
Link: https://www.cochrane.org/learn/courses-and-resources/cochrane-methodology/grade -
Source: cochrane.org
Title: thresholds and rating certainty of evidence using grade
Link: https://www.cochrane.org/events/thresholds-and-rating-certainty-of-evidence-using-gradeSource snippet
Decision thresholds guide judgments about...Read more...
-
Source: ftc.gov
Title: announces new business guidance marketers sellers health products
Link: https://www.ftc.gov/news-events/news/press-releases/2022/12/ftc-announces-new-business-guidance-marketers-sellers-health-productsSource snippet
Federal Trade CommissionFTC Announces New Business Guidance for Marketers...20 Dec 2022 — The Federal Trade Commission's Bureau of Consu...
-
Source: cochrane.org
Link: https://www.cochrane.org/learn/courses-and-resources/cochrane-methodology/grade-approach/grade-handbookSource snippet
GRADE HandbookIt incorporates the latest advancements, methodologies, and best practices, ensuring that users have access to practical gu...
-
Source: cochrane.org
Link: https://www.cochrane.org/events/introduction-assessing-certainty-evidence-gradeSource snippet
This Learning Live series is based on...Read more...
-
Source: cdc.gov
Link: https://www.cdc.gov/acip-grade-handbook/hcp/chapter-7-grade-criteria-determining-certainty-of-evidence/index.htmlSource snippet
Chapter 7: GRADE Criteria Determining Certainty of...22 Apr 2024 — This ACIP GRADE handbook provides guidance to the ACIP workgroups on...
-
Source: book.gradepro.org
Link: https://book.gradepro.org/guideline/overview-of-the-grade-approachSource snippet
of the GRADE approachGRADE offers methods, tools and processes to assess the certainty of evidence and support healthcare decision making...
-
Source: gradepro.org
Link: https://gradepro.org/handbook/Source snippet
GRADE handbookThe GRADE approach is a system for rating the quality of a body of evidence in systematic reviews and other evidence synthe...
-
Source: nhmrc.gov.au
Link: https://www.nhmrc.gov.au/file/23340/download?token=WCPzDa7mSource snippet
trength) of an evidence base as part of a systematic review.Read more...
Additional References
-
Source: jonesday.com
Link: https://www.jonesday.com/en/insights/2023/05/ftc-signals-intent-to-combat-deceptive-health-claims-advertisingSource snippet
FTC Targets Deceptive Health Claims AdvertisingFTC warns companies advertising health- and wellness-related products against making unsub...
-
Source: kslaw.com
Link: https://www.kslaw.com/attachments/000/010/346/original/It%27s_Not_Just_for_Dietary_Supplements_Anymore_-_FTC_Revises_and_Expands_Guidance_for_Health_Claims.pdf?1675367806=Source snippet
King & SpaldingFTC Revises and Expands Guidance for Health Claims2 Feb 2023 — According to the Guidance, surveys of individual experience...
-
Source: rivkinradler.com
Link: https://www.rivkinradler.com/publications/federal-court-rebuffs-ftc-effort-to-apply-pharmaceutical-standards-to-dietary-supplement-substantiation-2/Source snippet
Federal Court Rebuffs FTC Effort to Apply Pharmaceutical...Nov 3, 2015 — The Court's rejection of the FTC's efforts to require “drug” le...
-
Source: library-guides.imperial.ac.uk
Link: https://library-guides.imperial.ac.uk/systematic-review/gradeSource snippet
review: Summarising evidence certainty with GRADE7 days ago — It helps to move from evaluating evidence quality (methods) to evidence cer...
-
Source: kelleydrye.com
Link: https://www.kelleydrye.com/viewpoints/blogs/ad-law-access/misguided-the-ftc-attempts-to-redefine-the-law-with-its-health-products-compliance-guidanceSource snippet
Misguided: The FTC Attempts to Redefine the Law with its...21 Dec 2022 — Yesterday, the FTC's Bureau of Consumer Protection released its...
-
Source: fdli.org
Link: https://www.fdli.org/2023/02/ftc-and-nad-remind-industry-of-their-authority-over-all-health-product-advertising-including-rx/ -
Source: foodchainid.com
Link: https://www.foodchainid.com/resources/united-states-updated-ftc-guidance-on-claims-in-advertising/Source snippet
FTC Guidance on Advertising Claims UpdateUnderstand the latest Federal Trade Comission - FTC guidance on advertising claims, including su...
-
Source: crnusa.org
Title: crn ftc randomized clinical trials not required most supplements
Link: https://www.crnusa.org/newsroom/crn-ftc-randomized-clinical-trials-not-required-most-supplementsSource snippet
CRN to FTC: Randomized Clinical Trials Not Required...Sep 25, 2023 — “There is no legal requirement that a company possess randomized, c...
-
Source: cohenhealthcarelaw.com
Link: https://cohenhealthcarelaw.com/fda-ftc-law/advertising-and-marketing-claims/Source snippet
It means you must have solid scientific evidence—often including expert analysis or...Read more...
-
Source: cooley.com
Title: 2023 03 02 ftc revises health products compliance guidance
Link: https://www.cooley.com/news/insight/2023/2023-03-02-ftc-revises-health-products-compliance-guidanceSource snippet
FTC Revises Health Products Compliance Guidance2 Mar 2023 — The FTC's revised guidance applies to advertising for any health-related prod...
Topic Tree