UC Merced

Decision forests, also known as tree ensembles, have become an important model class in machine learning, particularly for tabular data, due to their excellent predictive accuracy, robustness, and minimal requirements for hyperparameter tuning. Remarkably, their widespread success has been achieved despite traditional training methods being largely heuristic, neither the individual trees nor the forest as a whole explicitly optimize a clearly defined loss function. Typically, these ensembles rely on methods such as bagging and boosting, where base learners are generated through greedy recursive partitioning. This dissertation proposes an alternative, optimization-driven framework for learning decision forests. Building upon the recently introduced Tree Alternating Optimization (TAO) algorithm, we first show significant improvements when explicitly optimized base learners replace the standard, heuristically constructed trees within two of the most popular boosting algorithms: AdaBoost and Gradient Boosting. Taking this idea further, we define an objective function over a fixed-size decision forest and extend alternating optimization to optimize the ensemble jointly, eliminating the greedy incremental approach of traditional boosting. The resulting method, Forest Alternating Optimization (FAO), is very effective at minimizing training error. Though they are more susceptible to overfitting, averaging multiple FAO forests achieves best generalization performance. Finally, we show the application of FAO to the special case of ensembles of decision stumps, which can naturally be viewed as generalized additive models (GAMs). By introducing targeted regularization strategies specifically designed for stump forests, we effectively mitigate overfitting, while consistently achieving state-of-the-art GAM performance.