UC San Diego

As computers are increasingly ubiquitous, it is becoming more important to make our interaction with them as easy and as efficient as possible. As alternatives to the standard duo: keyboards and mice, we have touch screens, handwriting recognition, voice recognition, gesture recognition, etc. The challenge of recognition systems is that they need to accurately interpret ambiguous signals, such as pen trajectories or speech, into discrete events, such as a sequence of letters. We call this task inference of user intent. Over time, as a particular user uses the interface, we would like performance of the inference to improve. There are two sides to this improvement. The first is that the user improves over time, as she learns how to speak or gesture in a less ambiguous way. The second is the adaptation of the computer, achieved through machine learning methods. We refer to this type of interaction as co-adaptation. This dissertation is a study of intent inference and of co-adaptation. The study is done in the context of two systems. The first system is a touch-free interface that is set up in a public space, called The Automatic Cameraman (TAC). On TAC, we focus on the problem of real-time user engagement. The second system is an adaptive handwriting recognition system for mobile phones, called uRight. The main characteristic of this system is that it not only adapts to each individual user but also provides insightful feedback to reinforce human learning. On this system, we explore the impact of co-adaptation on the information transfer rate in the context of handwriting recognition. Using machine learning to process and combine signals from two sensing modalities : visual and audio, TAC was able to robustly detect and track a user with minimal calibration. The engagement of the user was effectively identified by using a simple hand interaction protocol. Our experiment in co-adaptation confirmed our intuition that machine learning works best when matched with human adaptation. The concept of co- adaptation provides an interesting direction for future research on recognition systems