Donald Bren School of Information and Computer Sciences

This paper presents experimental results with both real and artificial data on using the technique of stacking to combine unsupervised learning algorithms. Specifically, stacking is used to form a linear combination of finite mixture model and kernel density estimators for non-parametric multivariate density estimation. The method is found to outperform other strategies such as choosing the single best model based on cross-validation, combining with uniform weights, and even using the single best model chosen by "cheating" and examining the test set. We also investigate in detail how the utility of stacking changes when one of the models being combined generated the data; how the stacking coefficients of the models compare to the relative frequencies with which cross-validation chooses among the models; how stacking performs using L1 and L2 performance measures (for which one must know the true density) rather than log-likelihood; visualization of combined "effective" kernels; and the sensitivity of stacking to overfitting as model complexity increases.