The striking thing about Table 1 is the large values
of the percentages of classification accuracy gained
by simple linear discriminant analysis. The lowest
percentage is 85% and in most cases over 90% of
the achievable improvement in predictive accuracy,
over the simple baseline model, is achieved by the
simple linear classifier.
I am grateful to Willi Sauerbrei for pointing out
that when the error rates of both the best method
and the linear method are small, the large proportion
in achievable accuracy which can be obtained
by the linear method corresponds to the error rate
of the linear method being a large multiple of that
of the best method. For example, in the most extreme
case in Table 1, the results for the segmentation
data show that the linear discrimination error
rate is nearly six times that of the best method. On
the other hand, when the error rates are small, this
large difference will correspond to only a small proportion
of new data points. Small differences in error
rate are susceptible to the issues raised in Sections 3
and 4: they may vanish when problem uncertainties
are taken into account.