Automatic Parameter Tuning
Many learning algorithms have parameters that can affect the outcome of learning.
For example, the decision tree learner C4.5 has two parameters that influence the
amount of pruning (we saw one, the minimum number of instances required in a
leaf, in Section 17.3). The k-nearest-neighbor classifier IBk has a parameter (k) that
sets the neighborhood size. But manually tweaking parameter settings is tedious,
just like manually selecting attributes, and presents the same problem: The test data
must not be used when selecting parameters; otherwise, the performance estimate
will be biased.