What Does "Unbiased Estimator" Mean? - Analytics

1. Learning center
2. Glossary

What does "Unbiased Estimator" mean?

Definition of Unbiased Estimator in the context of A/B testing (online controlled experiments).

What is an Unbiased Estimator?

Alias: unbiased

Finite-sample unbiasedness is one of the desirable properties of good estimators. An estimator is finite-sample unbiased when it does not show systemic bias away from the true value (θ*), on average, for any sample size n. If we perform infinitely many estimation procedures with a given sample size n, the arithmetic mean of the estimate from those will equal the true value θ*. In other words, the estimator's sampling distribution has a mean equal to the parameter it estimates.

It is easy to see why this is a desirable property: we do not want an estimator that systematically under or over-estimates the value. All estimators are subject to the bias-variance trade-off: the more unbiased an estimator is, the larger its variance, and vice-versa: the less variance it has, the more biased it becomes. A simple extreme example can be illustrate the issue. Say you are using the estimator E that produces the fixed value "5%" no matter what θ* is. Its variance is zero, however it is also maximally biased since it will show 5% no matter if the true value θ* is 3% or 99%.

Frequentist estimators used in A/B testing are normally unbiased. If not fully unbiased, then they aim to be close to achieving it. For example, in performing sequential testing one produces an estimator which is unconditionally-unbiased but which shows high bias in very early or very late monitoring stages. A part of an AGILE A/B test is the deployment of procedures that produce near-unbiased estimators conditional on the stopping stage.

Like this glossary entry? For an in-depth and comprehensive reading on A/B testing stats, check out the book "Statistical Methods in Online A/B Testing" by the author of this glossary, Georgi Georgiev.

Related A/B Testing terms

Consistent EstimatorEfficient EstimatorSufficient EstimatorEstimationMaximum Likelihood Estimate

About the author

Georgi Z. Georgiev

Georgi has over twenty years of experience in online marketing, web analytics, statistics, and design of business experiments.

Author of the book "Statistical Methods in Online A/B Testing", white papers on statistical analysis of A/B tests, and a speaker, he has been distinguished as a winner in the Data & Analytics category of the 2024 Experimentation Thought Leadership Awards.

Statistical Methods in Online A/B Testing

Take your A/B testing program to the next level with the most comprehensive book on user testing statistics in e-commerce.

Learn more

Glossary index by letter

ABCDEFGHIKLMNOPRSTUVZ

Select a letter to see all A/B testing terms starting with that letter or visit the Glossary homepage to see all.