For the depicted situation there are 2 methods which keep the nominal α value, namely
Kruskal-Wallis-test and Hotellings T 2 . All other kinds of analyses exceed the nominal
type I error rate by up to 8.6%. Especially F-test in analysis of variance and permutation
tests based on F-statistic perform very badly. Welch-test is within a 20% tolerance
(Rasch, Teuscher, & Guiard, 2007) and may be appropriate for this specific situation.
In Figure 2 the underlying situation is identical to that of Figure 1. In contrast to Figure 1
the nominal α value was defined as 0.01. Results are similar to that of Figure 1. Kruskal
Wallis test tends to be too conservative. Hotelling’s T 2 is the only method which keeps
the nominal α level exactly.