Automatically tracking people and their body poses in unconstrained videos is a core prob- lem of computer vision. It serves as a foundation for high-level reasoning such as activity recognition and human computer interaction. We consider two standard tracking tasks; tracking a human as its encapsulating bounding box or as an articulating body (poses).

Each task has its own challenge. The accuracy of tracking bounding boxes has been signifi- cantly improved for the past decade, but detecting small people remain challenging simply due to the lack of signals. The accuracy of tracking poses is noticeably lower, especially the one of tracking arms, mainly due to the fundamental difficulty of detecting indisticntive parts.

The algorithms for solving these problems are based on methodology of machine learning. A common pipeline is to project raw images to an invariant feature space, train a classifier (or regressor), and infer bounding boxes or poses from the trained model.

In this thesis, we aim to improve the accuracy in both tasks by proposing novel features, inference algorithms, and training schemes. In terms of tracking bounding boxes, we focus on multiresolution features and motion features that are aimed to robustly detect small people. In terms of tracking poses, we focus on combinatorial inference on part models and highly tuned appearance models.

We demonstrate our approaches using standard datasets and benchmarks of pedestrian de- tection and human pose estimation. Especially, our pedestrian detectors mark the top per- formance in Caltech Pedestrian Detection Benchmark among more than a dozen of recently developed detectors. We also achieve impressive performance in (upper body) pose estima- tion datasets.

Object detection is a central and challenging task in computer vision. In this thesis, we first examine the "big data" hypothesis: object detection might be solved with simple models backed with massive training data. We empirically show that the performance of one of the state-of-the-art methods (discriminatively trained HoG templates) tends to saturate fast when fed with more data. The required training data may need to grow exponentially in order to produce a fixed improvement in accuracy. We also find that the key difficulties in detection are large variation in object appearance and more importantly, that the variation exhibits a "long tail" distribution: there are many rare cases with little training data, which makes those cases hard to model. This thesis addresses such challenges by proposing new representations that share information within and across object subcategories. Sharing allows one to learn models for rare subcategories in the long-tail where traditional approaches suffer from lack of training data. We investigate two methods for sharing. We first examine global models that share entire training examples across multiple subcategories. For example, an SUV image might be used to train both a car and truck subcategory model. We also examine local sharing that share subwindows of training examples through "parts". For example, nearly all vehicles contain wheel parts. By mixing and matching (or composing) different parts together, one can implicitly encode an exponentially large set of subcategory models, which could even represent those subcategories not encountered in the training data.

We extensively experiment and evaluate our models on different benchmarks, and show superior performance over the state-of-the-art. Finally, we conclude with a detailed analysis of local part sharing for face analysis, perhaps the most well studied of all object recognition problems. By using semantically-defined parts (such as eyes, nose, lips), one can simultaneously perform face detection, pose estimation, and landmark localization with state-of-the-art accuracy, with a single model.

Robots are mechanically capable of doing many tasks, carrying loads, precisely manipulating objects, picking and packing or collaborating with humans. However, they require accurate 3D perception of objects and surrounding environment to do these tasks autonomously. Traditional methods build 3D representation of the scene using structure from motion techniques or depth sensors, while more recent approaches use statistical models to learn geometry and appearance of 3D objects and scenes. This thesis investigates approaches to represent, learn and analyze 3D objects in natural images. We first propose two new methods for 3D object recognition and pose estimation in single 2D images. Second, we study various geometric representations for the novel task of primitive 3D shape categorization.

We propose two novel approaches for recognizing 3D objects: (1) {\em Aligning a 3D model} to detected 2D landmarks, where we propose a novel method based on deformable-part models to propose candidate detections and 2D estimates of shape, then these estimates are refined by using an explicit 3D model of shape and viewpoint. (2) An {\em analysis by synthesis} approach where a forward synthesis model constructs possible geometric interpretations of the world, and then selects the interpretation that best agrees with the measured visual evidence. We show state of the art performance for detection and pose estimation on two challenging 3D object recognition datasets of cars and cuboids.

3D object recognition methods focus on modeling 3D shape of the objects, however, many objects may have similar 3D shape (washing machines, cabinets and microwave are all cuboidal), thus recognizing them require reasoning about appearance and geometry at the same time. The natural approach for recognition might extract pose-normalized appearance features. Though such approaches are extraordinarily common in the literature, in this thesis we demonstrate that they are {\em not optimal}. Instead, we introduce methods based on pose-synthesis, a somewhat simple approach of augmenting training data with geometrically perturbed training samples. We demonstrate that synthesis is a surprisingly simple but effective strategy that allows for state-of-the-art categorization and automatic 3D alignment.

In recent years, large-scale datasets with high-quality annotations have enabled many significant discoveries in computer vision and machine learning. However, the luxury of large-scale human annotations is not always affordable. For example, image semantic segmentation requires annotations for every pixel in an image. Similarly, action localization in untrimmed videos requires fine boundaries of action instances. The annotation effort for these tasks is prohibitively expensive to scale. What can we do in such scenarios? In this thesis, we look at different ways to circumvent this challenge. First, we examine micro-videos, a new and abundant source of visual data with user-generated metadata that could serve as web-supervision. Second, we explore weak supervision, in the context of action localization. While fine boundaries annotations of action instances are expensive and difficult to obtain, video-level labels can be cheaper and more accessible. We present two state-of-the-art systems that leveraged on this weaker form of supervision and compare them to their fully supervised counterparts. Lastly, we address the problems of accurately measuring the performance of computer vision systems with limited human annotations.

Cameras can naturally capture sequences of images, or videos, and for computers to understand videos, they must track to connect the past with the present. We focus on two problems which challenge current state-of-the-art trackers. First, we address the challenge of long-term occlusion. For this challenge, a tracker must know when it has lost track and how to reinitialize tracking when the target reappears. We tackle reinitialization by building good appearance models for humans and hands, with a particular emphasis on robustness and occlusion. For the second challenge, appearance variation, the tracker must know when and how to re-learn (or update) an appearance model. Common solutions to this challenge encounter the classic problem of drift: aggressively learning putative appearance changes allows small errors to compound, as elements of the background environment pollute the appearance model. We propose two solutions. First, we consider self-paced learning, wherein a tracker begins by learning from frames it finds easy. As the tracker becomes better at recognizing the target, it begins to learn from harder frames. We also develop a data-driven approach in which we train a tracking policy to decide when and how to update an appearance model. To take this direct approach to “learning when to learn”, we exploit large-scale Internet data through reinforcement learning. We interpret the resulting policy and conclude with extensions for tracking multiple objects. By solving these tracking challenges, we advance applications in augmented reality, vehicle automation, healthcare, and security.