Chapter 16 ANOVA Tables | Elements Of Statistical Modeling For ...

16.2 Example: a one-way ANOVA using the vole data

The vole data has a single factor (“treatment”) with three levels (“control”, “vitamin_E”, “vitamin_C”). In statistics textbooks that emphasize hypothesis testing, the “Which test should I use” flowchart would guide a researcher given this design to a single classification, or one-way ANOVA, since a t-test can only compare two levels but an ANOVA can compare more than two levels. There are better ways to think about what ANOVA is doing, but okay.

Here is an ANOVA table of the vole data:

I’ll explain all the parts of the ANOVA table later, but for now, focus on the \(p\)-value, which is that most researchers want out of the table. What null hypothesis does this \(p\)-value test? The p-value gives the probability of the observed \(F\) or larger \(F\), if the null were true. The null hypothesis models the data as if they were sampled from a single, normal distribution and randomly assigned to different groups. Thus the null hypotheis includes the equality of the means among factor levels. In the vole data, the single treatment factor has three levels and a small \(p\)-value could occur because of a difference in means between the vitamin_E treatment and control, or between the vitamin_C treatment and control, or between the two vitamin treatments. The \(p\)-value or ANOVA table doesn’t indicate what is different, only that the observed \(F\) is unexpectedly large if the null were true. As a consequence, researchers typically interpret a low \(p\)-value in an ANOVA table as evidence of “an effect” of the term but have to use additional tools to dissect this effect. The typical additional tools are either planned comparisons, which are contrasts among a subset of a priori identified treatment levels (or groups of levels) or unplanned comparisons (“post-hoc” tests) among all pairs of levels.

The \(p\)-value in the ANOVA table acts as a decision rule: if \(p < 0.05\) then it is okay to further dissect the factor with planned comparisons or post-hoc tests because the significant \(p\) “protects” the type I error of further comparisons. I’m not fond of using \(p\)-values for these sorts of decision rules.