1.3 - Unbiased Estimation | STAT 415 - STAT ONLINE

Skip to main content

Breadcrumb

1. Home
2. 1
3. 1.3

1.3 - Unbiased Estimation

On the previous page, we showed that if \(X_i\) are Bernoulli random variables with parameter \(p\), then:

\(\hat{p}=\dfrac{1}{n}\sum\limits_{i=1}^n X_i\)

is the maximum likelihood estimator of \(p\). And, if \(X_i\) are normally distributed random variables with mean \(\mu\) and variance \(\sigma^2\), then:

\(\hat{\mu}=\dfrac{\sum X_i}{n}=\bar{X}\) and \(\hat{\sigma}^2=\dfrac{\sum(X_i-\bar{X})^2}{n}\)

are the maximum likelihood estimators of \(\mu\) and \(\sigma^2\), respectively. A natural question then is whether or not these estimators are "good" in any sense. One measure of "good" is "unbiasedness."

Bias and Unbias Estimator

If the following holds:

\(E[u(X_1,X_2,\ldots,X_n)]=\theta\)

then the statistic \(u(X_1,X_2,\ldots,X_n)\) is an unbiased estimator of the parameter \(\theta\). Otherwise, \(u(X_1,X_2,\ldots,X_n)\) is a biased estimator of \(\theta\).

Example 1-4 Section

If \(X_i\) is a Bernoulli random variable with parameter \(p\), then:

\(\hat{p}=\dfrac{1}{n}\sum\limits_{i=1}^nX_i\)

is the maximum likelihood estimator (MLE) of \(p\). Is the MLE of \(p\) an unbiased estimator of \(p\)?

Answer

Recall that if \(X_i\) is a Bernoulli random variable with parameter \(p\), then \(E(X_i)=p\). Therefore:

\(E(\hat{p})=E\left(\dfrac{1}{n}\sum\limits_{i=1}^nX_i\right)=\dfrac{1}{n}\sum\limits_{i=1}^nE(X_i)=\dfrac{1}{n}\sum\limits_{i=1}^np=\dfrac{1}{n}(np)=p\)

The first equality holds because we've merely replaced \(\hat{p}\) with its definition. The second equality holds by the rules of expectation for a linear combination. The third equality holds because \(E(X_i)=p\). The fourth equality holds because when you add the value \(p\) up \(n\) times, you get \(np\). And, of course, the last equality is simple algebra.

In summary, we have shown that:

\(E(\hat{p})=p\)

Therefore, the maximum likelihood estimator is an unbiased estimator of \(p\).

Example 1-5 Section

If \(X_i\) are normally distributed random variables with mean \(\mu\) and variance \(\sigma^2\), then:

\(\hat{\mu}=\dfrac{\sum X_i}{n}=\bar{X}\) and \(\hat{\sigma}^2=\dfrac{\sum(X_i-\bar{X})^2}{n}\)

are the maximum likelihood estimators of \(\mu\) and \(\sigma^2\), respectively. Are the MLEs unbiased for their respective parameters?

Answer

Recall that if \(X_i\) is a normally distributed random variable with mean \(\mu\) and variance \(\sigma^2\), then \(E(X_i)=\mu\) and \(\text{Var}(X_i)=\sigma^2\). Therefore:

\(E(\bar{X})=E\left(\dfrac{1}{n}\sum\limits_{i=1}^nX_i\right)=\dfrac{1}{n}\sum\limits_{i=1}^nE(X_i)=\dfrac{1}{n}\sum\limits_{i=1}\mu=\dfrac{1}{n}(n\mu)=\mu\)

The first equality holds because we've merely replaced \(\bar{X}\) with its definition. Again, the second equality holds by the rules of expectation for a linear combination. The third equality holds because \(E(X_i)=\mu\). The fourth equality holds because when you add the value \(\mu\) up \(n\) times, you get \(n\mu\). And, of course, the last equality is simple algebra.

In summary, we have shown that:

\(E(\bar{X})=\mu\)

Therefore, the maximum likelihood estimator of \(\mu\) is unbiased. Now, let's check the maximum likelihood estimator of \(\sigma^2\). First, note that we can rewrite the formula for the MLE as:

\(\hat{\sigma}^2=\left(\dfrac{1}{n}\sum\limits_{i=1}^nX_i^2\right)-\bar{X}^2\)

because:

\(\displaystyle{\begin{aligned} \hat{\sigma}^{2}=\frac{1}{n} \sum_{i=1}^{n}\left(x_{i}-\bar{x}\right)^{2} &=\frac{1}{n} \sum_{i=1}^{n}\left(x_{i}^{2}-2 x_{i} \bar{x}+\bar{x}^{2}\right) \\ &=\frac{1}{n} \sum_{i=1}^{n} x_{i}^{2}-2 \bar{x} \cdot \color{blue}\underbrace{\color{black}\frac{1}{n} \sum x_{i}}_{\bar{x}} \color{black} + \frac{1}{\color{blue}\cancel{\color{black} n}}\left(\color{blue}\cancel{\color{black}n} \color{black}\bar{x}^{2}\right) \\ &=\frac{1}{n} \sum_{i=1}^{n} x_{i}^{2}-\bar{x}^{2} \end{aligned}}\)

Then, taking the expectation of the MLE, we get:

\(E(\hat{\sigma}^2)=\dfrac{(n-1)\sigma^2}{n}\)

as illustrated here:

\begin{align} E(\hat{\sigma}^2) &= E\left[\dfrac{1}{n}\sum\limits_{i=1}^nX_i^2-\bar{X}^2\right]=\left[\dfrac{1}{n}\sum\limits_{i=1}^nE(X_i^2)\right]-E(\bar{X}^2)\\ &= \dfrac{1}{n}\sum\limits_{i=1}^n(\sigma^2+\mu^2)-\left(\dfrac{\sigma^2}{n}+\mu^2\right)\\ &= \dfrac{1}{n}(n\sigma^2+n\mu^2)-\dfrac{\sigma^2}{n}-\mu^2\\ &= \sigma^2-\dfrac{\sigma^2}{n}=\dfrac{n\sigma^2-\sigma^2}{n}=\dfrac{(n-1)\sigma^2}{n}\\ \end{align}

The first equality holds from the rewritten form of the MLE. The second equality holds from the properties of expectation. The third equality holds from manipulating the alternative formulas for the variance, namely:

\(Var(X)=\sigma^2=E(X^2)-\mu^2\) and \(Var(\bar{X})=\dfrac{\sigma^2}{n}=E(\bar{X}^2)-\mu^2\)

The remaining equalities hold from simple algebraic manipulation. Now, because we have shown:

\(E(\hat{\sigma}^2) \neq \sigma^2\)

the maximum likelihood estimator of \(\sigma^2\) is a biased estimator.

Example 1-6 Section

If \(X_i\) are normally distributed random variables with mean \(\mu\) and variance \(\sigma^2\), what is an unbiased estimator of \(\sigma^2\)? Is \(S^2\) unbiased?

Answer

Recall that if \(X_i\) is a normally distributed random variable with mean \(\mu\) and variance \(\sigma^2\), then:

\(\dfrac{(n-1)S^2}{\sigma^2}\sim \chi^2_{n-1}\)

Also, recall that the expected value of a chi-square random variable is its degrees of freedom. That is, if:

\(X \sim \chi^2_{(r)}\)

then \(E(X)=r\). Therefore:

\(E(S^2)=E\left[\dfrac{\sigma^2}{n-1}\cdot \dfrac{(n-1)S^2}{\sigma^2}\right]=\dfrac{\sigma^2}{n-1} E\left[\dfrac{(n-1)S^2}{\sigma^2}\right]=\dfrac{\sigma^2}{n-1}\cdot (n-1)=\sigma^2\)

The first equality holds because we effectively multiplied the sample variance by 1. The second equality holds by the law of expectation that tells us we can pull a constant through the expectation. The third equality holds because of the two facts we recalled above. That is:

\(E\left[\dfrac{(n-1)S^2}{\sigma^2}\right]=n-1\)

And, the last equality is again simple algebra.

In summary, we have shown that, if \(X_i\) is a normally distributed random variable with mean \(\mu\) and variance \(\sigma^2\), then \(S^2\) is an unbiased estimator of \(\sigma^2\). It turns out, however, that \(S^2\) is always an unbiased estimator of \(\sigma^2\), that is, for any model, not just the normal model. (You'll be asked to show this in the homework.) And, although \(S^2\) is always an unbiased estimator of \(\sigma^2\), \(S\) is not an unbiased estimator of \(\sigma\). (You'll be asked to show this in the homework, too.)

Sometimes it is impossible to find maximum likelihood estimators in a convenient closed form. Instead, numerical methods must be used to maximize the likelihood function. In such cases, we might consider using an alternative method of finding estimators, such as the "method of moments." Let's go take a look at that method now.

• « Previous1.2 - Maximum Likelihood Estimation
• Next1.4 - Method of Moments »

Lesson

• Section 1: Estimation
  • Lesson 1: Point Estimation
    • 1.1 - Definitions
    • 1.2 - Maximum Likelihood Estimation
    • 1.3 - Unbiased Estimation
    • 1.4 - Method of Moments
  • Lesson 2: Confidence Intervals for One Mean
    • 2.1 - The Situation
    • 2.2 - A Z-Interval for a Mean
    • 2.3 - Interpretation
    • 2.4 - An Interval's Length
    • 2.5 - A t-Interval for a Mean
    • 2.6 - Non-normal Data
  • Lesson 3: Confidence Intervals for Two Means
    • 3.1 - Two-Sample Pooled t-Interval
    • 3.2 - Welch's t-Interval
    • 3.3 - Paired t-Interval
  • Lesson 4: Confidence Intervals for Variances
    • 4.1 - One Variance
    • 4.2 - The F-Distribution
    • 4.3 - Two Variances
  • Lesson 5: Confidence Intervals for Proportions
    • 5.1 - One Proportion
    • 5.2 - Two Proportions
  • Lesson 6: Sample Size
    • 6.1 - Estimating a Mean
    • 6.2 - Estimating a Proportion for a Large Population
    • 6.3 - Estimating a Proportion for a Small, Finite Population
  • Lesson 7: Simple Linear Regression
    • 7.1 - Types of Relationships
    • 7.2 - Least Squares: The Idea
    • 7.3 - Least Squares: The Theory
    • 7.4 - The Model
    • 7.5 - Confidence Intervals for Regression Parameters
    • 7.6 - Using Minitab to Lighten the Workload
  • Lesson 8: More Regression
    • 8.1 - A Confidence Interval for the Mean of Y
    • 8.2 - A Prediction Interval for a New Y
    • 8.3 - Using Minitab to Lighten the Workload

• Section 2: Hypothesis Testing
  • Lesson 9: Tests About Proportions
    • 9.1 - The Basic Idea
    • 9.2 - More Examples
    • 9.3 - The P-Value Approach
    • 9.4 - Comparing Two Proportions
    • 9.5 - Using Minitab
  • Lesson 10: Tests About One Mean
    • 10.1 - Z-Test: When Population Variance is Known
    • 10.2 - T-Test: When Population Variance is Unknown
    • 10.3 - Paired T-Test
    • 10.4 - Using Minitab
  • Lesson 11: Tests of the Equality of Two Means
    • 11.1 - When Population Variances Are Equal
    • 11.2 - When Population Variances Are Not Equal
    • 11.3 - Using Minitab
  • Lesson 12: Tests for Variances
    • 12.1 - One Variance
    • 12.2 - Two Variances
    • 12.3 - Using Minitab
  • Lesson 13: One-Factor Analysis of Variance
    • 13.1 - The Basic Idea
    • 13.2 - The ANOVA Table
    • 13.3 - Theoretical Results
    • 13.4 - Another Example
  • Lesson 14: Two-Factor Analysis of Variance
    • 14.1 - An Example
  • Lesson 15: Tests Concerning Regression and Correlation
    • 15.1 - A Test for the Slope
    • 15.2 - Three Tests for Rho
    • 15.3 - An Approximate Confidence Interval for Rho

• Section 3: Nonparametric Methods
  • Lesson 16: Chi-Square Goodness-of-Fit Tests
    • 16.1 - The General Approach
    • 16.2 - Extension to K Categories
    • 16.3 - Unspecified Probabilities
    • 16.4 - Continuous Random Variables
    • 16.5 - Using Minitab to Lighten the Workload
  • Lesson 17: Contingency Tables
    • 17.1 - Test For Homogeneity
    • 17.2 - Test for Independence
  • Lesson 18: Order Statistics
    • 18.1 - The Basics
    • 18.2 - The Probability Density Functions
    • 18.3 - Sample Percentiles
  • Lesson 19: Distribution-Free Confidence Intervals for Percentiles
    • 19.1 - For A Median
    • 19.2 - For Any Percentile
  • Lesson 20: The Wilcoxon Tests
    • 20.1 - The Sign Test for a Median
    • 20.2 - The Wilcoxon Signed Rank Test for a Median
    • 20.3 - Tied Observations
  • Lesson 21: Run Test and Test for Randomness
    • 21.1 - The Run Test
    • 21.2 - Test for Randomness
  • Lesson 22: Kolmogorov-Smirnov Goodness-of-Fit Test
    • 22.1 - The Test
    • 22.2 - Two Examples
    • 22.3 - A Confidence Band

• Section 4: Bayesian Methods
  • Lesson 23: Probability, Estimation, and Concepts
    • 23.1 - Subjective Probability
    • 23.2 - Bayesian Estimation

• Section 5: More Theory & Practice
  • Lesson 24: Sufficient Statistics
    • 24.1 - Definition of Sufficiency
    • 24.2 - Factorization Theorem
    • 24.3 - Exponential Form
    • 24.4 - Two or More Parameters
  • Lesson 25: Power of a Statistical Test
    • 25.1 - Definition of Power
    • 25.2 - Power Functions
    • 25.3 - Calculating Sample Size
  • Lesson 26: Best Critical Regions
    • 26.1 - Neyman-Pearson Lemma
    • 26.2 - Uniformly Most Powerful Tests
  • Lesson 27: Likelihood Ratio Tests
    • 27.1 - A Definition and Simple Example
    • 27.2 - The T-Test For One Mean
  • Lesson 28: Choosing Appropriate Statistical Methods
    • 28.1 - One Categorical Response
    • 28.2 - One Continuous Response
    • 28.3 - Two Continuous Measurements
    • 28.4 - Practice

× Save changes Close