Does the CRT Measure Reflection or Math?

Introduction

The Cognitive Reflection Test (CRT) is often presented as a measure of whether people can resist a tempting first answer and stop to think. That interpretation captures part of what the test does, but it is incomplete. The classic CRT questions also require basic numerical reasoning, arithmetic fluency and an understanding of quantitative relationships. As a result, a low score does not automatically mean that someone failed to reflect. In some cases, the person may have recognised that the intuitive answer was suspicious yet still lacked the numerical skills needed to reach the correct solution. Research over the past decade has increasingly treated CRT performance as a mixture of reflection and numeracy rather than a pure measure of reflective thinking alone.[Taylor & Francis Online+2ResearchGate]tandfonline.comTaylor & Francis OnlineThe cognitive reflection test revisited: exploring the ways…by B Szaszi · 2017 · Cited by 119 — Additionally, w…

Does the CRT Measure Reflection or Math?

Shane Frederick’s original CRT was designed to measure the tendency to override an intuitive but incorrect response and engage in further reflection. The famous bat-and-ball problem succeeds because it generates an answer that feels right before arithmetic verification occurs.[American Economic Association]aeaweb.orgAmerican Economic AssociationCognitive Reflection and Decision Makingby S Frederick · 2005 · Cited by 8284 — This paper introduces a thre…

The difficulty is that solving the problem correctly requires more than noticing the trap. A participant must also perform the relevant calculation. This creates an important measurement problem: the test cannot easily distinguish between someone who failed to reflect and someone who reflected but lacked the necessary numerical competence.

Researchers have repeatedly found substantial relationships between CRT scores and measures of numeracy, mathematical ability and quantitative reasoning. Some studies have argued that a significant portion of the CRT’s predictive power may come from its numerical demands rather than from reflection alone.[ResearchGate+2eScholarship]researchgate.net260061139 The Cognitive Reflection Test how much more than numerical abilityResearchGate(PDF) The Cognitive Reflection Test: how much more than…Frederick's (2005) Cognitive Reflection Test (CRT) is a 3-item tas…

This does not mean the CRT is “really just a maths test”. Rather, the evidence suggests that it blends at least two ingredients:

A person needs both to achieve a high score.[Taylor & Francis Online+2MOspace]tandfonline.comTaylor & Francis OnlineThe cognitive reflection test revisited: exploring the ways…by B Szaszi · 2017 · Cited by 119 — Additionally, w…

Why Arithmetic Skill Affects CRT Performance

The original CRT items appear simple, but each depends on quantitative concepts that not everyone handles equally well.

Consider the bat-and-ball problem. Someone may recognise that 10 cents cannot be correct because the numbers do not quite fit. Yet moving from that insight to the correct 5-cent answer requires forming and solving a simple numerical relationship. Similarly, the machine-and-widget problem depends on understanding rates, while the lily-pad problem requires reasoning about exponential growth rather than linear growth.

Researchers examining response patterns have found that errors do not always arise because participants blindly accept the intuitive answer. Some respondents reject the intuitive answer but still fail to reach the correct one. This suggests that inadequate numerical knowledge can be a separate source of failure.[Taylor & Francis Online]tandfonline.comTaylor & Francis OnlineThe cognitive reflection test revisited: exploring the ways…by B Szaszi · 2017 · Cited by 119 — Additionally, w…

A useful way to think about CRT performance is as a sequence:

Failure can occur at any stage. The final score alone does not reveal which stage caused the mistake.[Taylor & Francis Online]tandfonline.comAccording to the model of Individual Differences in Heuristics and Biases (Stanovich, K. E. (2018). Miserliness in human cognition: The…

What Numeracy Changes About Score Interpretation

The strongest critique of simplistic CRT interpretation is not that reflection is irrelevant. It is that reflection cannot be cleanly separated from numerical competence when using the classic items.

Several studies have concluded that CRT performance is best viewed as a multifaceted construct. Reflection matters, but so do numeracy and problem-solving ability. Research revisiting CRT performance has found evidence that both reflective tendencies and numerical skill independently contribute to scores.[Taylor & Francis Online+2PMC]tandfonline.comTaylor & Francis OnlineThe cognitive reflection test revisited: exploring the ways…by B Szaszi · 2017 · Cited by 119 — Additionally, w…

This changes how a low score should be interpreted.

A low score may indicate:

Likewise, a high score does not necessarily prove exceptional reflective habits. Strong numerical skills can make the correct answer easier to reach once reflection begins.[ResearchGate+2Edith Cowan University]researchgate.net260061139 The Cognitive Reflection Test how much more than numerical abilityResearchGate(PDF) The Cognitive Reflection Test: how much more than…Frederick's (2005) Cognitive Reflection Test (CRT) is a 3-item tas…

The practical lesson for improving analytical thinking is important. If someone performs poorly on CRT items, the next question should not be “Are they reflective?” but “What kind of difficulty produced the error?” The answer may point to reasoning habits, mathematical knowledge, or both.

Fairer Ways to Read a Wrong Answer

Looking only at whether an answer is correct throws away useful information.

Researchers studying CRT response patterns have noted that some participants generate neither the standard intuitive answer nor the correct answer. Others appear to reject the intuitive response but become stuck before reaching the solution. These patterns suggest that reflection and numeracy contribute in different ways.[Taylor & Francis Online]tandfonline.comTaylor & Francis OnlineThe cognitive reflection test revisited: exploring the ways…by B Szaszi · 2017 · Cited by 119 — Additionally, w…

A fairer interpretation of a wrong answer asks three questions:

Did the person immediately choose the intuitive answer?

If so, the error may reflect insufficient checking or reflection.

Did the person recognise the trap but fail to solve it?

That pattern points more towards a numeracy limitation than a reflection limitation.

Could the person explain why the intuitive answer was wrong?

Someone who can identify the flaw in the intuitive response may be demonstrating reflective thinking even if they cannot complete the calculation.

These distinctions matter because the educational response differs. Reflective weaknesses improve through habits of verification and deliberate checking. Numeracy weaknesses improve through practice with quantitative reasoning and mathematical concepts.

What the Evidence Means for Analytical Skill Development

The broader research record still supports the CRT as a valuable indicator of thinking style. Higher CRT scores predict performance on many judgement and decision-making tasks, and the test remains one of the most influential tools in the study of reasoning.[PubMed+2Springer]pubmed.ncbi.nlm.nih.govThe Cognitive Reflection Test as a predictor of…by ME Toplak · 2011 · Cited by 1580 — In this study, we showed that the CRT is a…

However, the evidence also warns against treating CRT scores as a pure measure of reflective thought. Studies examining the test’s validity increasingly conclude that successful performance depends on multiple capacities, including numeracy, insight and reflection.[ResearchGate+2Taylor & Francis Online]researchgate.netResearchGate(PDF) The Cognitive Reflection Test: A Measure of Intuition…19 Nov 2025 — The Cognitive Reflection Test (CRT) has increasi…

For anyone trying to improve analytical skills, the most useful takeaway is that reflection and numerical competence reinforce one another. Catching a first-answer trap requires the willingness to pause, but escaping it often requires the quantitative tools needed to evaluate alternatives. A wrong CRT answer therefore reveals less about a person’s overall reasoning quality than many popular discussions suggest. It may be evidence of a reflection problem, a numeracy problem, or a mixture of the two. Understanding that distinction leads to a more accurate and more constructive reading of the score.

Amazon book picks

Further Reading

Books and field guides related to Does the CRT Measure Reflection or Math?. Use these as the next step if you want deeper reading beyond the article.

Book

Thinking, Fast and Slow

By Daniel Kahneman

Directly addresses intuitive versus reflective thinking, the core debate behind CRT interpretation.

See on Amazon

Book

The Intelligence Trap

By David Robson

Explores why smart people make thinking errors and how reasoning skills differ from raw ability.

See on Amazon

Book

How Not to Be Wrong

By Jordan Ellenberg

Provides an accessible route into numeracy and quantitative reasoning, both highly relevant to CRT performance.

See on Amazon

Book

The Art of Thinking Clearly

By Rolf Dobelli

Covers common reasoning mistakes and judgment traps that resemble the intuitive errors CRT questions exploit.

See on Amazon

Browse more on Amazon:Thinking, Fast and SlowThe Intelligence TrapHow Not to Be Wrong

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Example eBay listing

Critical Thinking 2020 2 Movie Poster Art Print Display Famous Artwork StudioDec

SearcheBay.co.uk: critical thinking poster

Browse similar oneBay.co.uk

Example eBay listing

Critical Thinking 2020 4 Movie Poster Art Print PopCulture Wall Memorable Exclus

SearcheBay.co.uk: critical thinking poster

Browse similar oneBay.co.uk

Example eBay listing

Manic Street Preachers Signed Photo & CD Framed & COA Critical Thinking Poster

SearcheBay.co.uk: critical thinking poster

Browse similar oneBay.co.uk

Example eBay listing

Critical Thinking Framed Art Print Framed Wall Art Poster Canvas Print Picture

SearcheBay.co.uk: critical thinking poster

Browse similar oneBay.co.uk

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: researchgate.net
Title: 260061139 The Cognitive Reflection Test how much more than numerical ability
Link:https://www.researchgate.net/publication/260061139_The_Cognitive_Reflection_Test_how_much_more_than_numerical_ability

2. Source: pmc.ncbi.nlm.nih.gov
Title: PMCCommentary: Cognitive reflection vs
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC4722428/

3. Source: escholarship.org
Link:https://escholarship.org/uc/item/68n012fh

4. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC11152589/

5. Source: link.springer.com
Link:https://link.springer.com/article/10.3758/s13421

6. Source: researchgate.net
Link:https://www.researchgate.net/publication/51097609_The_Cognitive_Reflection_Test_as_a_Predictor_of_Performance_on_Heuristics-and-Biases_Tasks

7. Source: researchgate.net
Link:https://www.researchgate.net/publication/326134901_The_Cognitive_Reflection_Test_A_Measure_of_IntuitionReflection_Numeracy_and_Insight_Problem_Solving_and_the_Implications_for_Understanding_Real-World_Judgments_and_Beliefs

8. Source: escholarship.org
Title: What is the CRT?
Link:https://escholarship.org/content/qt9gp1r0wd/qt9gp1r0wd_noSplash_93e528871466d6184285f090a3111b91.pdf

9. Source: tandfonline.com
Link:https://www.tandfonline.com/doi/abs/10.1080/13546783.2017.1292954

10. Source: aeaweb.org
Link:https://www.aeaweb.org/articles?id=10.1257%2F089533005775196732

11. Source: ro.ecu.edu.au
Link:https://ro.ecu.edu.au/cgi/viewcontent.cgi?article=1797&context=ecuworks2013

12. Source: mospace.umsystem.edu
Link:https://mospace.umsystem.edu/bitstreams/878dd101-a441-4504-8eeb-b77e61fcca58/download

13. Source: tandfonline.com
Link:https://www.tandfonline.com/doi/full/10.1080/20445911.2025.2498573?src=exp-la

14. Source: pubmed.ncbi.nlm.nih.gov
Link:https://pubmed.ncbi.nlm.nih.gov/21541821/

15. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC5225989/

16. Source: Wikipedia
Title: Cognitive reflection test
Link:https://en.wikipedia.org/wiki/Cognitive_reflection_test

17. Source: ouci.dntb.gov.ua
Link:https://ouci.dntb.gov.ua/en/works/4OzGmgr7/

Additional References

18. Source: pureadmin.qub.ac.uk
Link:https://pureadmin.qub.ac.uk/ws/files/18636168/Primi_et_al._revised_03.04.15.pdf

19. Source: arno.uvt.nl
Link:https://arno.uvt.nl/show.cgi?fid=167379

20. Source: maggietoplak.com
Link:https://maggietoplak.com/wp-content/uploads/2025/05/Toplak-M.-E.-West-R.-F.-Stanovich-K.-E.-2014.-Rational-thinking-and-cognitive-sophistication-Development-cognitive-abilities-and-thinking-dispositions.pdf

21. Source: philpapers.org
Link:https://philpapers.org/rec/TOPAMI

22. Source: youtube.com
Link:https://www.youtube.com/watch?v=BNeR1snRBuA

23. Source: youtube.com
Link:https://www.youtube.com/watch?v=VJ4D1XJbkbo

24. Source: youtube.com
Link:https://www.youtube.com/watch?v=K0O1_nwAky0

25. Source: cambridge.org
Link:https://www.cambridge.org/core/journals/judgment-and-decision-making/article/performance-on-the-cognitive-reflection-test-is-stable-across-time/43048174A721F456121196EF46EA1A7F

26. Source: cambridge.org
Link:https://www.cambridge.org/core/journals/judgment-and-decision-making/article/reflection-on-cognitive-reflection-testing-convergentdivergent-validity-of-two-measures-of-cognitive-reflection/C3EAFF6628EC678F4FF234967B95F665

27. Source: law.yale.edu
Title: Frederick Cognitive Reflectionand Decision Making
Link:https://law.yale.edu/sites/default/files/area/workshop/leo/document/Frederick_CognitiveReflectionandDecisionMaking.pdf