A Simple Example
Statistical modeling is a sequential process in which
one gradually refines the model to provide a better
and better fit to the distributions from which the
data were drawn. In general, the earlier stages in this
process yield greater improvement in model fit than
later stages. Furthermore, if one looks at the historical
development of classification methods, then the
earlier approaches involve relatively simple structures
(e.g., the linear forms of linear or logistic discriminant
analysis), while more recent approaches
involve more complicated structures (e.g., the decision
surfaces of neural networks or support vector
machines). It follows that the simple approaches will
have led to greater improvement in predictive performance
than the later approaches which are necessarily
trying to improve on the predictive performance
obtained by the simpler earlier methods. Put
another way, there is a law of diminishing returns.
Although this paper is concerned with supervised
classification problems, it is illuminating to examin