A lot of different situations in regard to factor levels, variance ratios and sample sizes
were examined. Table 1 shows results for a range of factor levels between 3 and 20. The
number of observations changed from 3 to 20 with equal sample sizes. The ratio of standard
deviations was fixed to 2:1:…:1. Hotelling’s T 2 can only be used in situations,
where the number of observations per sample is at least the number of factor levels. This
is caused by the fact, that no program was available for calculating the cumulative density
function of Hotelling’s T 2 directly. Therefore the approximation to F distribution
was used. For the specific test this approximation restricts the number of factor levels to
the number of observations at the most. With the availability of tables for Hotelling’s T 2
this limitation will fall.
In Table 1 simulation results (at a nominal α value of 0.01) for all 7 methods are presented.
The number of factor levels ranged from 3 to 20. The design was a balanced one,
with a number of observations which ranged from 3 to 20, too.
Table 1 shows, that even in situations with a narrow ratio of standard deviations F-test in
analysis of variance as well as permutation tests based on F-statistic don’t keep nominal
type I error rate. Welch test performs very badly in situations when the number of factor
levels is high, especially with small sample sizes. Weighted Analysis of Variance (SAS
Mixed) is also not appropriate in most situations. Kruskal Wallis test is very conservative