Within Calibration
Why Your Safe Range May Be Too Tight
Interval estimates reveal overprecision when supposedly safe ranges miss the truth more often than they should.
On this page
- What an interval estimate is supposed to mean
- How narrow ranges reveal overprecision
- Ways to widen ranges without becoming vague
Page outline Jump by section
Introduction
A confidence interval is meant to express uncertainty, not just a best guess. If you say you are 90% confident that the answer lies between two values, then, over many similar judgements, the true answer should fall inside that range about nine times out of ten. When those supposedly safe ranges miss the truth much more often than expected, the intervals are too narrow. This is one of the clearest signs of overprecision: being excessively certain about the limits of what you know rather than necessarily being wrong about the central estimate itself. [PMC]pmc.ncbi.nlm.nih.govPMCA simple remedy for overprecision in judgmentNIHby U Haran · 2010 · Cited by 185 — Overprecision is typically measured by eliciting a confidence interval — a range of values th…
For improving thinking and analytical skills, this distinction matters. A person can repeatedly produce reasonable best guesses while still making poor decisions because they underestimate how much uncertainty surrounds those guesses. Narrow intervals encourage risk-taking, premature commitment and surprise when reality falls outside an unrealistically tight range.
What an interval estimate is supposed to mean
An interval estimate combines two ideas:
- A point estimate: your best guess.
- A range: how uncertain you believe that guess is.
Suppose you estimate that a project’s cost will be £95,000. The point estimate is £95,000. A 90% interval might be £85,000 to £110,000. The important claim is not that every value inside the range is equally likely. Instead, the claim is that your process for producing such intervals should capture the truth roughly 90% of the time.
This is known as coverage. If you later examine one hundred comparable estimates labelled “90% confidence”, approximately ninety should contain the true value. If only sixty do, your intervals are substantially too narrow, regardless of how reasonable your central estimates appeared. [PMC]pmc.ncbi.nlm.nih.govPMCA simple remedy for overprecision in judgmentNIHby U Haran · 2010 · Cited by 185 — Overprecision is typically measured by eliciting a confidence interval — a range of values th…
How narrow ranges reveal overprecision
Overprecision differs from simply making inaccurate predictions. It is specifically about underestimating uncertainty.
Imagine two analysts predicting annual sales:
- Analyst A predicts £10 million with a range of £9.9–10.1 million.
- Analyst B predicts £10 million with a range of £8.8–11.2 million.
If actual sales are £10.6 million, both analysts missed with their point estimate. However, only Analyst B expressed uncertainty realistically enough for the outcome to remain plausible.
This matters because decision-makers often act on the apparent certainty of an estimate rather than its exact value. A narrow range suggests that contingency plans are unnecessary, while a wider but well-calibrated range communicates that important surprises remain possible.
Classic studies beginning with Alpert and Raiffa found that when people were asked for 90% confidence intervals on general knowledge questions, their intervals commonly captured the correct answer only around 30–60% of the time. The phenomenon has been replicated repeatedly across occupations, cultures and levels of expertise. [learnmoore.org]learnmoore.organd you will get hit rates between 30% and 60%, suggesting they have drawn…Read more…
Why people make intervals too tight
Several psychological mechanisms push intervals towards excessive precision.
Anchoring on the first estimate. People typically generate a best guess first, then adjust outward only slightly. The adjustments rarely go far enough to represent genuine uncertainty.
Difficulty imagining unknown possibilities. Most people think primarily about reasons their estimate is correct instead of systematically searching for reasons it could be wrong. Alternative scenarios receive too little weight.
Illusion of knowledge. Having some information often creates a stronger feeling of certainty than is justified. Familiarity with a topic may narrow subjective uncertainty faster than objective uncertainty.
Preference for precision. Precise answers appear more competent and useful than vague ones. As a result, people often produce intervals that communicate confidence rather than honestly representing uncertainty.
Research has also shown that many respondents appear not to interpret “90% confidence interval” literally. They often provide intervals much narrower than genuine 90% beliefs would require, suggesting that they balance informativeness against accuracy rather than reporting their true uncertainty. [PMC+2Semantic Scholar]pmc.ncbi.nlm.nih.govPMCA simple remedy for overprecision in judgmentNIHby U Haran · 2010 · Cited by 185 — Overprecision is typically measured by eliciting a confidence interval — a range of values th…
A useful warning sign: repeated misses
One narrow interval proves very little. Even a perfectly calibrated 90% interval will occasionally fail.
The warning sign is a consistent pattern.
For example, suppose over twenty forecasts labelled “90% confidence”:
- the true value falls inside only 11 intervals;
- nine miss entirely.
That is not simply bad luck. It indicates that the uncertainty estimates are systematically underestimated.
The same principle applies in business forecasting, project planning, scientific prediction and everyday judgement. Whenever “high-confidence” estimates fail far more often than their stated confidence implies, the problem is usually not just poor prediction but poor calibration.
Professional forecasting surveys exhibit similar behaviour. Even trained forecasters often produce prediction intervals whose observed coverage falls well below their stated confidence levels, illustrating that expertise alone does not eliminate overprecision. [University of California Press]online.ucpress.eduOverprecision in the Survey of ProfessionalInside 90% confidence intervals, the hit rate is 66.41%.Read more…
Ways to widen ranges without becoming vague
The goal is not to make intervals enormous. An interval stretching from almost any value to almost any other value is well calibrated but nearly useless. Better calibration means widening intervals only enough to match reality.
Several practical techniques help.
Ask what would have to happen for your estimate to fail. Instead of asking why your estimate is right, deliberately search for plausible reasons it could be too high or too low.
Estimate lower and upper limits separately. Rather than expanding around a single anchor, independently ask:
- What value would surprise me if reality were lower?
- What value would surprise me if reality were higher?
This reduces the influence of the initial point estimate.
Review past misses. Compare previous intervals with actual outcomes. Notice whether surprises tend to occur in one direction or both.
Use outside information. Before relying on detailed knowledge of the current case, examine what similar situations have historically produced. Base rates often reveal wider uncertainty than intuition alone suggests.
Practise with feedback. Calibration improves when people repeatedly make interval estimates, receive objective outcomes and adjust future ranges accordingly. Simply making more predictions without feedback produces much less improvement. [PMC]pmc.ncbi.nlm.nih.govPMCA simple remedy for overprecision in judgmentNIHby U Haran · 2010 · Cited by 185 — Overprecision is typically measured by eliciting a confidence interval — a range of values th…
The balance between useful precision and honest uncertainty
An effective interval is neither maximally narrow nor maximally wide. It communicates the greatest amount of useful information while still achieving its advertised coverage.
This creates an unavoidable trade-off:
- Narrower intervals are more informative but risk frequent failure.
- Wider intervals capture reality more reliably but become less useful for decision-making.
Good judgement lies in finding the balance where intervals are as tight as the evidence genuinely permits—not as tight as confidence or appearance encourages.
Recognising intervals that are too narrow therefore strengthens analytical thinking in a specific way. It shifts attention away from defending a favourite estimate and towards representing uncertainty faithfully. That makes forecasts more trustworthy, decisions more resilient and unexpected outcomes less surprising when reality inevitably proves more complex than our first impression.
eBay marketplace picks
Marketplace Samples
Live-tested eBay searches with available results related to this page.
Endnotes
-
Source: pmc.ncbi.nlm.nih.gov
Title: PMCA simple remedy for overprecision in judgment
Link: https://pmc.ncbi.nlm.nih.gov/articles/PMC5386407/Source snippet
NIHby U Haran · 2010 · Cited by 185 — Overprecision is typically measured by eliciting a confidence interval — a range of values th...
-
Source: learnmoore.org
Link: https://learnmoore.org/mooredata/HOC.pdfSource snippet
and you will get hit rates between 30% and 60%, suggesting they have drawn...Read more...
-
Source: semanticscholar.org
Link: https://www.semanticscholar.org/paper/Overconfidence-in-interval-estimates.-Soll-Klayman/991dbb96824914065de380d6c7e5d589532fde9eSource snippet
We report five experiments showing that people have much less confidence...Read more...
-
Source: online.ucpress.edu
Title: Overprecision in the Survey of Professional
Link: https://online.ucpress.edu/collabra/article/10/1/92953/200113/Overprecision-in-the-Survey-of-ProfessionalSource snippet
Inside 90% confidence intervals, the hit rate is 66.41%.Read more...
Additional References
-
Source: d-nb.info
Link: https://d-nb.info/1248484126/34Source snippet
s for a given confidence level α, simply by taking an interval in the output space that contains α of...Read more...
-
Source: researchgate.net
Link: https://www.researchgate.net/publication/255968087_True_Overconfidence_in_Interval_Estimates_Evidence_Based_on_a_New_Measure_of_MiscalibrationSource snippet
Often less than 50% of the true values fall within 90% confidence intervals, indicating...Read more...
-
Source: kevindorst.substack.com
Title: The Case for Overconfidence is Overstated
Link: https://kevindorst.substack.com/p/the-case-for-overconfidence-is-overstatedSource snippet
by Kevin DorstYour 90%-confidence intervals are calibrated if 90% of them contain the true value.1 They are overprecise if less than 90%...
-
Source: youtube.com
Title: Overconfidence: The Most Dangerous Bias in Decision Making
Link: https://www.youtube.com/watch?v=ei_zP4saqPsSource snippet
How to (and how not to) Interpret Confidence Intervals - YouTube How to (and how not to) Interpret Confidence Intervals - YouTube...
-
Source: ohdsi.github.io
Title: Empirical Ci Calibration Vignette
Link: https://ohdsi.github.io/EmpiricalCalibration/articles/EmpiricalCiCalibrationVignette.htmlSource snippet
Empirical calibration of confidence intervals14 Feb 2025 — In this document we use an example study to illustrate how CIs can be calibrat...
-
Source: youtube.com
Title: Overconfidence Bias: Why We Overestimate Our Financial Knowledge
Link: https://www.youtube.com/watch?v=Gcqbb5eDB0wSource snippet
How to (and how not to) Interpret Confidence Intervals...
-
Source: youtube.com
Title: How to (and how not to) Interpret Confidence Intervals
Link: https://www.youtube.com/watch?v=FQAcHqgMKMISource snippet
Overconfidence: The Most Dangerous Bias in Decision Making...
-
Source: youtube.com
Title: Why People Are So Confident When They’re Wrong
Link: https://www.youtube.com/watch?v=9M_QK4stCJUSource snippet
Overconfidence Bias: Why We Overestimate Our Financial Knowledge...
-
Source: youtube.com
Title: Overconfidence Bias | Psychology of Decision-Making
Link: https://www.youtube.com/watch?v=eq6SCQNQNscSource snippet
Why People Are So Confident When They're Wrong...
-
Source: arxiv.org
Link: https://arxiv.org/pdf/2510.26995Source snippet
more...
Topic Tree