Cross-validated likelihood is investigated as a tool for automatically determining the appropriate number of components (given the data) in finite mixture modelling, particularly in the context of model-based probabilistic clustering. The conceptual framework for the cross-validation approach to model selection is direct in the sense that models are judged directly on their out-of-sample predictive performance. The method is applied to a well-known clustering problem in the atmospheric science literature using historical records of upper atmosphere geopotential height in the Northern hemisphere. Cross-validated likelihood provides strong evidence for three clusters in the data set, providing an objective confirmation of earlier results derived using non-probabilistic clustering techniques.

This paper presents experimental results with both real and artificial data on using the technique of stacking to combine unsupervised learning algorithms. Specifically, stacking is used to form a linear combination of finite mixture model and kernel density estimators for non-parametric multivariate density estimation. The method is found to outperform other strategies such as choosing the single best model based on cross-validation, combining with uniform weights, and even using the single best model chosen by "cheating" and examining the test set. We also investigate in detail how the utility of stacking changes when one of the models being combined generated the data; how the stacking coefficients of the models compare to the relative frequencies with which cross-validation chooses among the models; how stacking performs using L1 and L2 performance measures (for which one must know the true density) rather than log-likelihood; visualization of combined "effective" kernels; and the sensitivity of stacking to overfitting as model complexity increases.

Machine learning and data mining have the potential to provide meaningful solutions to many real-world problems that could impact society. This dissertation explores machine learning methods that utilize the sequential or temporal aspect of data to solve socially-relevant problems in two domains: one in education, the other in medicine.

In the domain of education and learning analytics, I present two different methods that help better understand students' online learning behaviors based on their clickstream data. First, statistical change detection is used to detect when and how students change their behavior during a course relative to the student population as a whole. I group the students depending on the type of changes and examine how the changes can be related to the course outcomes. The second method I explore is the use of probabilistic clustering to allow different types of temporal behavioral patterns to emerge from clickstream data. The resulting patterns are analyzed in the context of procrastination and time-management, demonstrating that the procrastinating behavior and course outcomes are highly correlated.

Secondly, in the medical domain, different materials and methods are discussed to test the feasibility of using machine learning models to obtain structured information from patient-physician conversations in primary care visits, as a means to potentially address the physician burnout problem. Several hundred dialog transcripts of doctor-patient conversations are used to predict topic labels for talk-turns. Different machine learning models are trained to operate on single or multiple talk-turns (logistic classifiers, support vector machines, gated recurrent units [GRUs]) as well as sequential models that integrate information across talk-turn sequences (conditional random fields, hidden Markov models, and hierarchical GRUs). From the results, I show that incorporating sequential information across talk-turns improves the accuracy of topic prediction in dialogs. Moreover, I examine the degree to which topic classification accuracy drops by adding an automatic speech recognition (ASR) system for transcription to the pipeline. A systematic evaluation is carried out by measuring the performance of ASR (with word error rate) and the downstream classification accuracy.

In the second part of my work in the medical domain, I investigate the use of machine learning methods for predicting emotional valence of both doctors and patients at the utterance level, based on transcripts of doctor-patient visits. This is one of the first large-scale investigations of emotional valence prediction at the utterance level in long medical dialogs. Using a variety of evaluation metrics, I show that current machine learning methods can achieve accuracies for this task that are close to that of human performance.

This paper addresses the problem of clustering data when the available data measurements are not multivariate vectors of fixed dimensionality. For example, one might have data from a set of medical patients, where for each patient there are time series, image, text, and multivariate data. We propose a general probabilistic clustering framework for clustering heterogeneous data types of this form. We focus on two-level probabilistic hierarchical models, consisting of a high-level mixture model on parameters and a low-level model for observations. This general framework permits probabilistic clustering of "objects" (sequences, histograms, images, etc) using an extension of the expectation-maximization (EM) algorithm which we derive. We further show that earlier (intuitive) clustering algorithms can be viewed as special cases (approximations) of the framework proposed here. The paper includes several illustrations of the method, including an application to a problem in clustering two-dimensional histograms of red blood cell data in a medical diagnosis context.

Personalized recommendation is an important, yet challenging task that benefits both service provides and consumers, by enabling more efficient and informative choices.

However, despite the increased number of input streams, the user-generated data are sparse, since time constrains limit the number of items humans can interact with. Additionally, people make item selections in a variety of contexts, which can significantly affect their consumption behavior. This thesis is focused on exploring such data, and introducing models that can address these issues.

For the dissertation’s first contribution, we focus on data that include consumption of both new and old items. There has been significant prior work on developing predictive modeling techniques for recommending \emph{new} items to individuals, however there are many situations where making predictions for both previously-consumed and new items for an individual is of interest. We develop a mixture model framework that addresses sparsity constrains through empirical Bayesian priors, and can balance individual preferences in terms of exploration and exploitation. We evaluate our model using several real-world datasets, including location, social media, and music listening data.

Next, we investigate the problem of incorporating contextual information when making predictions. We focus on the problem of language modeling, where context (such as the identity of the speaker) can alter language use. We propose a general approach to language modeling that can efficiently and dynamically incorporate multiple types of external contextual information, without increasing the complexity of the model. Experiments on Reddit and Yelp corpora demonstrate that the proposed approach not only results in increased accuracy over competing methods, but also provides useful insights into how different contexts affect language use.

In the final chapter we investigate a more practical matter, and present a method that can generate more detailed affinity data between users and attributes. There are a lot of available data, where users have written a review of an entity and given it a score. However, this score is general, and does not represent how the reviewer feels about each attribute of the entity.

In this chapter we take on the problem of learning classifiers from group labels that are able to make predictions at the instance level.

We evaluate our approach using three large review data sets from IMDB, Amazon and Yelp, and demonstrate that the approach is both accurate and scalable compared to various alternatives.

In this paper, the technique of stacking, previously only used for supervised learning, is applied to unsupervised learning. Specifically, it is used for non-parametric multivariate density estimation, to combine finite mixture model and kernel density estimators. Experimental results on both simulated data and real world data sets clearly demonstrate that stacked density estimation outperforms other strategies such as choosing the single best model based on cross-validation, combining with uniform weights, and even the single best model chosen by "cheating" by looking at the data used for independent testing.