Multiple testing
Some students will try to perform a large number of tests on their data. The chance of a
type I error increases with the number of tests. Adjustments to keep the type I error low
for a larger number of tests are included as post hoc tests in ANOVA. This will mean less of
the results are statistically significant. The most commonly used post hoc tests are Tukey
and Sidak although Scheffe’s is often used in medicine.
Suggest that the student looks in their notes or papers in their field when choosing. If
adjustments need to be made by hand, the Bonferroni adjustment is the easiest to explain
although it is the most conservative (least likely to lead to rejection of the null). Either
divide the significance level initially used (probably 0.05) by the number of tests being
carried out and compare the p-value with the new, smaller significance
level. Alternatively, multiply the p-value by the number of tests being carried out and
compare to 0.05. This is the standard adjustment made after the Kruskall-Wallis and has a
maximum limit of 1 (as it’s a probability!).