In this chapter we describe new experiments with the ensemble learning method
Stacking. The central question in these experiments was whether meta-learning
methods can be used to accurately predict various aspects of Stacking’s behaviour.
The resulting contributions of this chapter are two-fold: When learning
to predict the accuracy of Stacking, we found that the single most important
feature is the accuracy of the best base classifier. A simple linear model involving
just this feature turns out to be surprisingly accurate. When learning
to predict significant differences between Stacking and three related ensemble
learning methods, we have found simple models, all but one of which are based
on single features which can be efficiently computed directly from the dataset.
For one of these models, we were able to offer a tentative interpretation. These
models may ultimately be used to decide in advance which ensemble learning
scheme to use on a given dataset, since neither of them is always the best choice.
Furthermore, aiming to understand these models can lead to new insights into
Stacking’s behaviour. This chapter is an extended version of (Seewald, 2002b).