Ensembles
The characteristic feature of an ensemble is redundancy; each predictor could perform the
task on its own, but better generalization performance is obtained from the combination.
The idea of an ensemble of neural networks can be traced back to Nilsson [99] who arranged
a group of perceptrons in a layer, and combined their outputs with a second, vote-taking,
layer. In this section we will review Bagging and Boosting, two of the most widely used
techniques for creating an ensemble.