Does the App Raise Scores, or Track Motivation?

Introduction

Students who use revision apps often achieve higher exam scores than students who do not. At first glance, this seems to show that the app improves learning. However, this is a classic example of why analytical thinking distinguishes correlation from causation. The students who voluntarily download, configure and consistently use revision tools may already differ from other students in important ways. They may be more motivated, more organised, more interested in high achievement, or have stronger study habits before they ever open the app. Unless those differences are accounted for, higher scores among app users cannot automatically be credited to the software itself. Research on educational technology repeatedly highlights this challenge, while also showing that some app features—such as spaced repetition and retrieval practice—can genuinely improve learning when tested under stronger research designs. Journal of Learning Analytics+2bera-journals.onlinelibrary.wiley.com[learning-analytics.info]learning-analytics.infoAssessment & Evaluation in Higher Education…Read more…

The difference between app use and app effect

Imagine a school finds that pupils using a revision app score 10% higher on average than those who do not. That observation alone answers only one question: who uses the app? It does not answer the more important question: what would have happened if the same pupils had not used it?

The distinction matters because educational technology is rarely assigned at random. Students usually choose whether to install an app, how often to use it, and whether to continue after the novelty wears off. Those decisions are themselves influenced by characteristics that also affect academic performance.

A motivated student is more likely to:

Each of these behaviours can raise examination performance independently of the app. If motivation explains both app use and higher marks, motivation becomes a confounding factor rather than the app being the sole cause of better results. This is precisely the kind of selection bias that causal inference seeks to identify and control. Journal of Learning Analytics+2University of Colorado Boulder[learning-analytics.info]learning-analytics.infoAssessment & Evaluation in Higher Education…Read more…

Why motivation is such a powerful confounder

Motivation is difficult to observe directly, yet it influences almost every stage of learning. Students who are intrinsically motivated often study more frequently, monitor their own understanding, ask for feedback, and revise over longer periods rather than relying on last-minute cramming. Those same students are also more likely to experiment with new educational tools.

This creates a misleading pattern:

The same issue appears in many educational settings. Students who attend optional revision sessions, visit office hours or complete practice quizzes usually outperform their peers, but attendance itself may simply indicate stronger engagement rather than being the entire reason for improved outcomes. Without careful comparison, it is impossible to separate the effect of the intervention from the characteristics of the students choosing it.[Journal of Learning Analytics+2PMC]learning-analytics.infoAssessment & Evaluation in Higher Education…Read more…

Confounders in student performance data

Motivation is only one possible explanation. Several other factors may create the appearance that a revision app is highly effective.

Prior attainment. Students who already perform well may adopt sophisticated study tools earlier than others.

Self-regulated learning. Learners who routinely plan study sessions, monitor progress and review mistakes often use educational apps more consistently.

Time spent studying. App users may simply spend more total time revising, regardless of whether the app itself is especially effective.

Socioeconomic factors. Access to reliable devices, internet connections and quiet study environments can influence both app usage and examination results.

Teacher encouragement. Schools or teachers who actively recommend particular revision platforms may also provide better overall academic support.

These variables can overlap, making simple comparisons between users and non-users increasingly unreliable. Learning analytics researchers have repeatedly warned that observational educational data can easily confuse engagement with intervention effects if these competing explanations are ignored.[Journal of Learning Analytics]learning-analytics.infoAssessment & Evaluation in Higher Education…Read more…

When higher scores may genuinely reflect app benefits

Recognising confounding does not mean revision apps are ineffective. Many incorporate learning techniques with strong independent evidence.

One widely supported example is spaced repetition, where material is reviewed at increasing intervals instead of repeatedly in one sitting. Another is retrieval practice, where learners actively recall information through testing rather than simply rereading notes. Both approaches have extensive cognitive psychology evidence supporting improved long-term retention.

If a revision app successfully encourages students to use these evidence-based techniques more consistently than they otherwise would, then the app can have a genuine causal effect. In that case, the mechanism is not “using an app” in itself, but the learning behaviours the app promotes. Studies evaluating spaced-repetition systems have found improved knowledge retention and examination performance under controlled or quasi-experimental conditions, although results vary by subject, implementation and learner population.[PMC+2ResearchGate]pmc.ncbi.nlm.nih.govImplementation of a spaced-repetition approach to enhance…by K Vagha · 2025 · Cited by 5 — In conclusion, this study highlights the…

This distinction illustrates an important analytical habit: instead of asking whether technology works, ask which specific features change learning behaviour and through what mechanism.

What stronger education evidence would compare

The strongest evidence tries to estimate what would have happened if otherwise similar students had revised differently.

Several research designs are more convincing than simple observational comparisons.

Even these methods have limitations, but they substantially reduce the risk of attributing existing differences between students to the technology itself. Modern educational and causal inference research increasingly emphasises these approaches because straightforward comparisons of users and non-users often exaggerate intervention effects. Journal of Learning Analytics+2Life Sciences Education[learning-analytics.info]learning-analytics.infoAssessment & Evaluation in Higher Education…Read more…

A practical way to interpret claims about revision apps

When you encounter claims such as “students using this revision app scored 15% higher”, treat them as the beginning of an investigation rather than the end.

Ask questions such as:

These questions move attention from surface correlations towards causal explanations.

The broader lesson for analytical thinking

Revision apps provide a useful illustration of a wider principle in reasoning. High-performing people are often early adopters of productive tools, making the tools appear more powerful than they really are. Sometimes the tool genuinely improves outcomes; sometimes it mainly attracts people who would have performed well anyway.

Better analytical thinking requires separating who chooses an intervention from what the intervention actually changes. Only by identifying and testing rival explanations—especially motivation, prior ability and study habits—can we judge whether a revision app truly raises scores or merely tracks the kinds of students most likely to succeed.

Amazon book picks

Further Reading

Books and field guides related to Does the App Raise Scores, or Track Motivation?. Use these as the next step if you want deeper reading beyond the article.

Book

Make It Stick

By Peter C. Brown, Henry L. Roediger III et al.

Provides evidence-based explanations of learning methods such as retrieval practice that are discussed in the article.

See on Amazon

Book

The Art of Statistics

By David Spiegelhalter

Explains how to interpret educational data, confounding variables, and causal claims.

See on Amazon

Book

How to Lie with Statistics

By Darrell Huff

Shows how misleading conclusions can arise from observational data and poor statistical reasoning.

See on Amazon

Book

Naked Statistics

By Charles Wheelan

Builds intuition about evidence, comparisons, and interpreting claims about educational outcomes.

See on Amazon

Browse more on Amazon: Make It Stick The Art of Statistics How to Lie with Statistics

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Shop location

UsingUSA USA USAUKCanadaAustraliaIreland

Example eBay listing

Wooden Office Study Desk Pen Pencil Holder Storage Organizer Box Tidy Case Stand

SearcheBay.co.uk: study desk organizer

Browse similar oneBay.co.uk

Example eBay listing

Office Study Desk Organizer Table Box Tidy Case Wooden Pen Pencil Storage Holder

SearcheBay.co.uk: study desk organizer

Browse similar oneBay.co.uk

Example eBay listing

Office Study Desk Wooden Pen Pencil Storage Holder Organizer Box Tidy Shelf Case

SearcheBay.co.uk: study desk organizer

Browse similar oneBay.co.uk

Example eBay listing

Office Study Desk Organizer Table Box Tidy Case Display Stationaries Pen Holder

SearcheBay.co.uk: study desk organizer

Browse similar oneBay.co.uk

Browse more oneBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

1. Source: learning-analytics.info
Link:https://learning-analytics.info/index.php/JLA/article/view/7577

2. Source: bera-journals.onlinelibrary.wiley.com
Link:https://bera-journals.onlinelibrary.wiley.com/doi/10.1111/bjet.13116

3. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC12343689/

4. Source: colorado.edu
Link:https://www.colorado.edu/education/sites/default/files/attached-files/Briggs_Causal%20Inference%20and%20the%20Heckman%20Model.pdf

5. Source: pmc.ncbi.nlm.nih.gov
Title: PMCSelection Effects and Prevention Program Outcomes
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC3760982/

6. Source: researchgate.net
Link:https://www.researchgate.net/publication/290511665_Spaced_Repetition_Promotes_Efficient_and_Effective_Learning_Policy_Implications_for_Instruction

7. Source: strong.io
Link:https://www.strong.io/blog/causal-inference-methods-a-survey

8. Source: youtube.com
Title: Spaced repetition in learning theory
Link:https://www.youtube.com/watch?v=cVf38y07cfk

9. Source: youtube.com
Title: Selection Bias
Link:https://www.youtube.com/watch?v=dJaHpN2L_C4

10. Source: lifescied.org
Link:https://www.lifescied.org/doi/10.1187/cbe.25

11. Source: stratfordjournalpublishers.org
Link:https://stratfordjournalpublishers.org/journals/index.php/journal-of-education/article/download/1712/2254/5498

12. Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC12040101/

13. Source: wol.iza.org
Link:https://wol.iza.org/articles/machine-learning-for-causal-inference-in-economics/long

Additional References

14. Source: arxiv.org
Link:https://arxiv.org/abs/2312.15042

15. Source: arxiv.org
Link:https://arxiv.org/abs/2210.04552

16. Source: aera.net
Link:https://www.aera.net/portals/38/docs/causal%20effects.pdf

17. Source: ijcsrr.org
Link:https://ijcsrr.org/wp-content/uploads/2025/07/43-1707-2025.pdf

18. Source: youtube.com
Title: Clinical Research and Statistics
Link:https://www.youtube.com/watch?v=DmyBTe6NKuo

19. Source: educationendowmentfoundation.org.uk
Title: effectiveness of edtech reflections from new review
Link:https://educationendowmentfoundation.org.uk/news/effectiveness-of-edtech-reflections-from-new-review

20. Source: discovery.ucl.ac.uk
Title: Outhwaite Educational apps and learning AAM
Link:https://discovery.ucl.ac.uk/10173200/1/Outhwaite_Educational%20apps%20and%20learning_AAM.pdf

21. Source: methods.sagepub.com
Title: propensity score methods and applications
Link:https://methods.sagepub.com/book/mono/preview/propensity-score-methods-and-applications.pdf

22. Source: selfdeterminationtheory.org
Title: 2024 DavidWeinstein TechInClass
Link:https://selfdeterminationtheory.org/wp-content/uploads/2024/06/2024_DavidWeinstein_TechInClass.pdf

23. Source: eprints.whiterose.ac.uk
Link:https://eprints.whiterose.ac.uk/id/eprint/234521/1/29_01_10125-73656.pdf