We present a new ensemble learning algorithm,DeepBoost, which can use as base classifiers a
hypothesis set containing deep decision trees, ormembers of other rich or complex families, and
succeed in achieving high accuracy without overfitting the data. The key to the success of the algorithm is a capacity-conscious criterion for the selection of the hypotheses. We give new datadependent learning bounds for convex ensembles expressed in terms of the Rademacher complexities of the sub-families composing the base classifierset, and the mixture weight assigned to eachsub-family. Our algorithm directly benefits from these guarantees since it seeks to minimize the
corresponding learning bound. We give a full description of our algorithm, including the details
of its derivation, and report the results of severalexperiments showing that its performance compares favorably to that of AdaBoost and Logistic Regression and their L1-regularized variants.