Within Practice Tests
Can You Use the Idea Somewhere New?
Practice testing is strongest when questions make you use an idea in a changed situation.
On this page
- Why recall is not enough for analysis
- Examples of transfer questions for thinking tools
- How to test whether an idea fits a case
Page outline Jump by section
Introduction
Practice testing becomes much more valuable when it asks a simple question: can you use this idea somewhere new? Remembering a definition is useful, but durable understanding means recognising the same principle when the surface details have changed. Someone who truly understands confirmation bias should notice it in a political debate, a business meeting and their own decision-making—not only in the textbook example where they first encountered it.
Research on retrieval practice increasingly shows that testing is not just about remembering facts. Well-designed retrieval questions can improve the ability to transfer knowledge to unfamiliar situations, especially when learners repeatedly apply concepts to varied examples rather than recalling the same example over and over. [Andy Matuschak+2PubMed]andymatuschak.orgMore specifically, 4 experiments examined how repeated…Read more…
Why recall is not enough for analysis
Analytical thinking depends on recognising underlying structure rather than superficial appearance. Two situations may look different while sharing the same principle, or they may look similar while requiring different reasoning.
Simple recall questions test whether information can be reproduced:
- What is the definition of opportunity cost?
- What is a confounding variable?
- What is survivorship bias?
These questions matter because ideas must first be available from memory. However, they do not reveal whether someone knows when those ideas apply.
Transfer questions ask something different:
- Which idea helps explain this new situation?
- Does the concept still apply after important details have changed?
- What would have to be true before you could use this principle confidently?
These questions require learners to map an abstract concept onto a fresh problem instead of matching familiar wording.
Educational psychologists often distinguish between recognising an answer and generating one. Recognition can be supported by familiarity. Transfer requires selecting an appropriate mental model without obvious cues, making it a much stronger test of conceptual understanding. [PubMed]pubmed.ncbi.nlm.nih.govThe critical role of retrieval practice in long-term retentionby HL Roediger 3rd · 2011 · Cited by 2694 — In addition, retrieval pr…
Examples of transfer questions for thinking tools
A useful transfer question changes the context while preserving the underlying reasoning.
Instead of asking:
What is confirmation bias?
ask:
A manager dismisses evidence that contradicts their preferred hiring candidate. Which thinking error might explain this behaviour?
Instead of asking:
Define base rates.
ask:
A new medical test is advertised as 99% accurate. What information would you want before deciding how convincing a positive result really is?
Instead of asking:
What is opportunity cost?
ask:
Your team decides to spend six months improving an existing product. What important alternative uses of those resources should also be considered?
Instead of asking:
What is correlation?
ask:
A city reports that neighbourhoods with more parks also have healthier residents. Does that alone justify concluding that parks caused better health?
Notice that none of these questions asks learners to repeat definitions. Each requires identifying whether a familiar analytical tool fits an unfamiliar scenario.
Why varying examples strengthens transfer
One common mistake is practising the same type of example repeatedly. Learners become good at recognising that particular example rather than understanding the underlying idea.
Research suggests that retrieval becomes more transferable when people repeatedly retrieve and apply concepts across different examples instead of revisiting the same one. In a series of experiments, participants who answered application questions using varied examples later performed better on completely new examples than participants who repeatedly practised the same example. [PubMed]pubmed.ncbi.nlm.nih.govRetrieving and applying knowledge to different examples…by AC Butler · 2017 · Cited by 87 — All four experiments showed that var…
The likely reason is that varied practice encourages abstraction. Instead of remembering one story, learners begin identifying the deeper pattern shared across many situations.
For analytical thinking, this is especially valuable because real-world problems rarely resemble textbook cases exactly. Decisions in finance, healthcare, engineering or public policy often differ in surface details while sharing similar logical structure.
How to test whether an idea fits a case
Transfer questions work best when they force deliberate judgement rather than automatic pattern matching.
A practical sequence is:
- Describe the situation without naming the concept.
- Ask which analytical idea, if any, applies.
- Require an explanation of why it fits.
- Ask what evidence would make the idea inappropriate instead.
The fourth step is often overlooked. Good thinkers do not merely recognise similarities; they also recognise limits.
For example, instead of asking whether confirmation bias is present, ask:
- What evidence supports that interpretation?
- What alternative explanation might better fit?
- What facts would change your conclusion?
This shifts retrieval from labelling towards evaluation.
Near transfer and far transfer
Not all transfer is equally demanding.
Near transfer involves situations that differ only modestly from practice. A statistical concept learned in one research paper is applied to another research paper with a similar design.
Far transfer requires applying the same principle in a substantially different context. A lesson about evaluating scientific evidence might later help someone assess claims in advertising, investment decisions or public debate.
Far transfer has historically been difficult to achieve in education. Nevertheless, retrieval practice has shown encouraging effects. Andrew Butler’s experiments demonstrated that repeated testing produced better transfer to new inferential questions than repeated studying, including transfer across different knowledge domains. Later work replicated this advantage while exploring the conditions under which it occurs. [Andy Matuschak+2PMC]andymatuschak.orgMore specifically, 4 experiments examined how repeated…Read more…
The implication is not that any quiz automatically produces broad reasoning skills. Rather, the design of retrieval questions matters. Questions that demand explanation, inference and application are more likely to support flexible knowledge than questions that only require verbatim recall.
Common mistakes when writing transfer questions
Many questions appear novel but actually test memory in disguise.
Weak transfer questions often:
- Change only names while leaving the structure identical.
- Include obvious vocabulary that reveals the correct concept.
- Ask for definitions after presenting the answer.
- Reward memorised phrases rather than reasoning.
Stronger questions instead:
- Remove familiar labels.
- Introduce realistic distractions.
- Require choosing between several plausible concepts.
- Ask learners to justify why one idea fits better than another.
The goal is not to make questions artificially difficult but to make them resemble the uncertainty found in genuine thinking.
A practical habit for building durable understanding
After studying any important idea, ask yourself three questions before moving on:
- Where else could this principle apply?
- Where would it fail to apply?
- How would I recognise it if nobody used its technical name?
These questions encourage abstraction rather than memorisation. They also expose shallow understanding early, while there is still time to strengthen it.
Practice testing reaches its greatest value when retrieval becomes more than remembering yesterday’s example. By repeatedly asking learners to recognise familiar principles inside unfamiliar situations, transfer questions help transform isolated knowledge into thinking tools that remain useful long after the original lesson has been forgotten. [PDF Retrieval Practice+3PubMed+3Andy Matuschak]pubmed.ncbi.nlm.nih.govRetrieving and applying knowledge to different examples…by AC Butler · 2017 · Cited by 87 — All four experiments showed that var…
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: andymatuschak.org
Link: https://andymatuschak.org/files/papers/Butler%20-%202010%20-%20Repeated%20Testing%20Produces%20Superior%20Transfer%20of%20Learning%20Relative%20to%20Repeated.pdfSource snippet
More specifically, 4 experiments examined how repeated...Read more...
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/29265856/Source snippet
Retrieving and applying knowledge to different examples...by AC Butler · 2017 · Cited by 87 — All four experiments showed that var...
-
Source: pubmed.ncbi.nlm.nih.gov
Link: https://pubmed.ncbi.nlm.nih.gov/20951630/Source snippet
The critical role of retrieval practice in long-term retentionby HL Roediger 3rd · 2011 · Cited by 2694 — In addition, retrieval pr...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC5183614/Source snippet
The Testing Effect and Far Transfer: The Role of Exposure to...by GG van Eersel · 2016 · Cited by 31 — Butler (2010: Experiment 3) sh...
-
Source: pdf.retrievalpractice.org
Link: https://pdf.retrievalpractice.org/transfer/Pan_Rickard_2018.pdfSource snippet
PDF Retrieval PracticeTransfer of Test-Enhanced Learning: Meta-Analytic Review...by SC Pan · 2018 · Cited by 433 — The critical question...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC12292765/Source snippet
Use of Retrieval Practice in the Health Professions - PMCby MJ Serra · 2025 · Cited by 11 — Most empirical articles examining retrieval p...
-
Source: pmc.ncbi.nlm.nih.gov
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC3983480/Source snippet
practice enhances new learning: the forward effect of...by B Pastötter · 2014 · Cited by 276 — The review discusses current theoretical...
-
Source: notes.andymatuschak.org
Title: Retrieval practice and transfer learning
Link: https://notes.andymatuschak.org/Retrieval_practice_and_transfer_learningSource snippet
practice and transfer learning16 Apr 2024 — Butler, A. (2010). Repeated Testing Produces Superior Transfer of Learning Relative to Repeat...
Additional References
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/40869048_Tests_Enhance_the_Transfer_of_LearningSource snippet
(PDF) Tests Enhance the Transfer of LearningPrior studies suggest that a greater degree of transfer reduces the size of the testing effec...
-
Source: scribd.com
Link: https://www.scribd.com/document/594207830/Butler-2010-Repeated-Testing-Produces-Superior-Transfer-of-Learning-Relative-to-Repeated-StudyingSource snippet
Butler 2010 - Repeated Testing Produces Superior...Butler 2010 - Repeated Testing Produces Superior Transfer of Learning Relative to Rep...
-
Source: sc-pan.github.io
Link: https://sc-pan.github.io/pdf/PR_JEPA_2017.pdfSource snippet
C. (2010). Repeated testing produces superior transfer of learn- ing relative to repeated studying. Journal of Experimental Psychology: L...
-
Source: researchgate.net
Title: Repeated Testing Produces Superior Transfer of Learning
Link: https://www.researchgate.net/publication/46094320_Repeated_Testing_Produces_Superior_Transfer_of_Learning_Relative_to_Repeated_StudyingSource snippet
transfer of knowledge in a variety of contexts (Butler, 2010).... retrieval practice may also promote transfer of learning to new clini...
-
Source: scribd.com
Link: https://www.scribd.com/document/986681099/Elicit-Retrieval-Practice-and-Memory-Transfer-ReportSource snippet
rm retention and knowledge transfer across various academic domains...Read more...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/321988375_Retrieving_and_applying_knowledge_to_different_examples_promotes_transfer_of_learningSource snippet
ings, one of the most significant goals of learning in HE (Butler 2010...
-
Source: my.chartered.college
Title: Lots of potential for using the same
Link: https://my.chartered.college/impact_article/[assessmentSource snippet
as learning: The role of retrieval practice in...by J Firth · Cited by 4 — Butler, AC (2010) Repeated testing produces superior transfer...
-
Source: frontiersin.org
Link: https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2016.01977/fullSource snippet
The Testing Effect and Far Transfer: The Role of Exposure...by GG van Eersel · 2016 · Cited by 32 — Butler (2010: Experiment 3) showed t...
-
Source: uweb.cas.usf.edu
Title: Rohrer et al 2010JEPLMC
Link: https://uweb.cas.usf.edu/~drohrer/pdfs/Rohrer_et_al_2010JEPLMC.pdfSource snippet
Enhance the Transfer of Learning - Karst Research Groupby D Rohrer · 2010 · Cited by 374 — Studies have shown that material is better rem...
-
Source: tandfonline.com
Link: https://www.tandfonline.com/doi/full/10.1080/00220973.2024.2392684Source snippet
Retrieval Practice: A Tool for Teaching the Control-of-...by J Kranz · 2024 · Cited by 2 — Two experiments investigated through a transf...
Topic Tree