Within Causation
Did Onboarding Help, or Did Motivated Customers Join?
Customers who attend onboarding may spend more because they were already more motivated, not because onboarding changed them.
On this page
- Why voluntary users are not a fair comparison
- Signals that selection bias is driving the result
- Better tests for onboarding impact
Page outline Jump by section
Introduction
Customers who attend onboarding sessions often spend more money than those who do not. At first glance, this appears to show that onboarding increases customer value. However, this conclusion can be misleading because the people who choose to attend onboarding are frequently different from those who do not. They may already be more motivated, have larger budgets, be implementing the product for more important business needs, or simply be more willing to invest time in learning it. In other words, the observed difference may reflect selection bias rather than the effect of onboarding itself.
Recognising this distinction is an important analytical skill. Rather than asking whether onboarding attendees spend more, the better question is: Would those same customers have spent more even if they had never attended onboarding? Until that question is addressed, a correlation should not be treated as evidence of causation. Research on causal inference consistently identifies self-selection as a major threat to drawing reliable conclusions from observational business data.[ResearchGate]researchgate.netResearch Gate(PDF) Making Stronger Causal InferencesResearchGate(PDF) Making Stronger Causal InferencesMay 17, 2018 — We develop competing hypotheses about the relationship between high per…
Why voluntary users are not a fair comparison
Voluntary onboarding creates two groups that differ before the onboarding session begins.
Customers who sign up for onboarding are often those who:
- have already decided to make significant use of the product;
- are implementing it across larger teams;
- have executive support or larger budgets;
- are more engaged during the trial or purchasing process;
- are willing to invest time because they expect substantial value.
Customers who skip onboarding may include casual users, organisations with limited resources, or experienced users who believe they need little assistance. These pre-existing differences influence future spending regardless of whether onboarding is effective.
Suppose a software company finds that onboarding attendees spend twice as much during their first year. That observation alone cannot distinguish between two very different explanations:
- onboarding taught customers how to use advanced features, increasing purchases; or
- customers planning major deployments were always more likely to attend onboarding and spend more.
Both explanations fit the same dashboard. Without additional evidence, the data cannot determine which is correct.
This is a classic example of selection bias, where the characteristics influencing participation are also related to the outcome being measured. Studies across business and organisational research show that failing to account for selection effects can substantially exaggerate or even reverse apparent treatment effects.[ResearchGate]researchgate.netResearch Gate(PDF) Making Stronger Causal InferencesResearchGate(PDF) Making Stronger Causal InferencesMay 17, 2018 — We develop competing hypotheses about the relationship between high per…
Signals that selection bias is driving the result
Several warning signs suggest that higher spending among onboarding participants may reflect customer differences rather than onboarding success.
Attendance is voluntary. When customers decide for themselves whether to participate, motivation becomes part of the comparison. Highly motivated customers tend to engage with many product resources, not only onboarding.
High-value customers receive more encouragement. Customer success teams frequently prioritise larger accounts for personalised onboarding. If premium customers are deliberately offered more support, higher future revenue is partly built into the process.
Pre-onboarding differences already exist. If onboarding participants already have higher contract values, larger organisations, more active users, or greater feature usage before onboarding begins, comparing later spending without adjustment is misleading.
Engagement moves together. Customers who attend onboarding often also open more emails, contact support, complete setup tasks, and participate in webinars. Onboarding may simply be one visible part of a broader pattern of engagement.
The effect appears immediately. If attendees already have higher spending commitments before the onboarding session occurs, the apparent benefit cannot reasonably be attributed to training received afterwards.
Looking for these patterns helps analysts identify situations where selection is likely influencing the results.
Better tests for onboarding impact
The strongest evidence comes from comparing customers who are similar before onboarding but differ only in whether they receive it.
Several approaches are commonly used.
Randomised experiments
The most reliable approach is to randomly assign eligible customers to different onboarding experiences. Randomisation makes the groups similar on average before the intervention, allowing later differences in spending to be interpreted more confidently as causal effects.
Natural experiments
Sometimes operational constraints effectively create random variation. For example, customers might receive onboarding only when consultant capacity is available or because of scheduling differences unrelated to customer quality. Such situations can sometimes approximate experimental conditions if carefully evaluated.
Matched comparisons
When experiments are impossible, analysts can compare customers with similar characteristics, including:
- company size;
- initial purchase value;
- industry;
- product usage before onboarding;
- acquisition channel;
- customer tenure.
Methods such as propensity score matching or covariate balancing attempt to reduce differences between groups before estimating onboarding effects. While these techniques cannot eliminate hidden differences, they usually provide more credible estimates than simple before-and-after comparisons.[ResearchGate]researchgate.netResearch Gate(PDF) Making Stronger Causal InferencesResearchGate(PDF) Making Stronger Causal InferencesMay 17, 2018 — We develop competing hypotheses about the relationship between high per…
Regression with appropriate controls
Statistical models can adjust for measurable differences such as account size or initial engagement. However, they only account for variables that are actually measured. Unobserved factors—such as customer enthusiasm or internal organisational priorities—may still bias estimates.
A practical dashboard example
Imagine a software company reports:
- Onboarding attendees spend an average of £9,000 during their first year.
- Non-attendees spend an average of £5,000.
The immediate conclusion might be that onboarding increases spending by £4,000.
A closer investigation finds that onboarding attendees also:
- purchased larger initial licences;
- invited more users during the first week;
- completed setup more quickly;
- were more likely to request enterprise features before onboarding.
Those findings suggest that onboarding attendees already represented stronger customers before the programme started.
After matching customers with similar initial characteristics, the estimated difference might fall substantially. Perhaps onboarding still improves spending—but by hundreds of pounds rather than thousands. In some cases, the apparent advantage may disappear entirely once selection bias is addressed.
The key lesson is not that onboarding is ineffective. Rather, it is that the raw comparison was answering the wrong question.
Thinking more carefully about business metrics
Business dashboards often encourage quick causal stories because they display simple comparisons between customer groups. Yet these groups frequently differ long before a product feature, marketing campaign, or customer success programme is introduced.
When evaluating onboarding results, useful questions include:
- Were customers choosing whether to participate?
- Were high-value customers more likely to receive invitations?
- Did attendees already differ before onboarding?
- What evidence rules out pre-existing motivation as the explanation?
- Would similar customers who missed onboarding have behaved differently?
These questions shift attention from observed associations to plausible causal explanations. They help analysts avoid rewarding programmes simply because they attract the customers who were already most likely to succeed, leading to more reliable decisions about which initiatives genuinely improve customer outcomes.
Amazon book picks
Further Reading
Books and field guides related to Did Onboarding Help, or Did Motivated Customers Join?. Use these as the next step if you want deeper reading beyond the article.
The Book of Why
Explains why correlation alone cannot establish causation and how to reason about causal effects.
Mostly Harmless Econometrics
Introduces practical approaches to dealing with self-selection and identifying causal relationships.
Causal Inference
Covers selection bias, observational data, and methods for estimating causal effects.
How to Measure Anything
Encourages rigorous business measurement and evidence-based evaluation instead of misleading comparisons.
eBay marketplace picks
Marketplace Samples
Example marketplace items related to this page. Use the search link to explore similar finds on eBay.
Endnotes
1.
Source: researchgate.net
Title: Research Gate(PDF) Making Stronger Causal Inferences
Link:https://www.researchgate.net/publication/325209736_Making_Stronger_Causal_Inferences_Accounting_for_Selection_Bias_in_Associations_Between_High_Performance_Work_Systems_Leadership_and_Employee_and_Customer_Satisfaction
Source snippet
ResearchGate(PDF) Making Stronger Causal InferencesMay 17, 2018 — We develop competing hypotheses about the relationship between high per...
Published: May 17, 2018
2.
Source: youtube.com
Title: Selection Bias: A Threat to Internal Validity
Link:https://www.youtube.com/watch?v=d1MKMlHi-uc
Source snippet
Causal Inference - EXPLAINED...
3.
Source: youtube.com
Title: Causal Inference
Link:https://www.youtube.com/watch?v=Od6oAz1Op2k
Source snippet
Selection Bias causal inference economics econometrics The 6 Minute Review Guide on Selection Bias, ATT, and ATE In Case of Econ Struggles...
Additional References
4.
Source: dataford.io
Link:https://dataford.io/questions/causal-onboarding-impact-on-retention
Source snippet
Causal Onboarding Impact on RetentionScenario You changed the onboarding flow and saw better retention afterward. The concern is that the...
5.
Source: lunduniversity.lu.se
Link:https://www.lunduniversity.lu.se/lup/publication/9048129
Source snippet
qualitative case study on how the onboarding process...The purpose of this study is to to investigate how cultural control can be integr...
6.
Source: chartis-research.com
Link:https://www.chartis-research.com/market-analysis/7821011/effective-onboarding-in-wholesale-banks-examining-challenges-and-defining-best-practices
Source snippet
Effective Onboarding in Wholesale Banks12 Apr 2021 — This collaborative report from Chartis and iMeta discusses the risk and technology p...
7.
Source: pubmed.ncbi.nlm.nih.gov
Title: Pub Med Volunteerism and self-selection bias in human positron
Link:https://pubmed.ncbi.nlm.nih.gov/23196924/
Source snippet
Scientists have known for decades that persons who volunteer for behavioral research may be different from those who decline participatio...
8.
Source: select-statistics.co.uk
Title: accounting for self selection biases in customer satisfaction surveys
Link:https://select-statistics.co.uk/case-studies/accounting-for-self-selection-biases-in-customer-satisfaction-surveys/
Source snippet
Accounting for Self-Selection Biases in Customer...23 Oct 2012 — The Challenge Customer Satisfaction Surveys (CSS's) are a useful tool f...
9.
Source: medium.com
Link:https://medium.com/design-bootcamp/crafting-a-self-serving-onboarding-experience-for-content-creators-fed4f5ef0e0c
Source snippet
customers' sign-up time based on collaborative research.Read more...
10.
Source: youtube.com
Title: Best Tips To Avoid Biases in Business Intelligence | Google Career Certificates
Link:https://www.youtube.com/watch?v=Q6fvryMZmYw
Source snippet
The 6 Minute Review Guide on Selection Bias, ATT, and ATE...
11.
Source: pmc.ncbi.nlm.nih.gov
Link:https://pmc.ncbi.nlm.nih.gov/articles/PMC12456455/
Source snippet
population-based investigation of participation rate and self...by AA Stone · 2023 · Cited by 95 — These results suggest that participan...
12.
Source: youtube.com
Title: Tackling Selection Bias in Marketing Models
Link:https://www.youtube.com/watch?v=O0xuPSnAPa4
Source snippet
Best Tips To Avoid Biases in Business Intelligence | Google Career Certificates...
13.
Source: arno.uvt.nl
Link:https://arno.uvt.nl/show.cgi?fid=162863
Source snippet
Onboarding Success Factors Based On Employee...13 Jan 2023 — Employees in the present study are anticipated to discuss how communication...
Topic Tree