Reconstruction happens in the human brain every day. When humans watch their surrounding scene, they effortlessly infer dynamic representations of scene geometry from sequences of images. This higher dimensional reconstruction not only helps to interpret the input data but also provides an important basis for performing complex, higher level tasks. In spite of the importance of scene reconstruction, it has been considered a difficult task since the amount of information in the input image (2D) is insufficient to fully reconstruct scene geometry (3D). Performing such a task clearly requires the use of prior knowledge. In this thesis, we explore the advantages of machine learning-based techniques in order to reconstruct geometric information from images or videos. We utilize deep neural networks and probabilistic models and demonstrate their effectiveness in reconstructing geometric information.

As the first part of this thesis, we estimate 2D motion flow from video, leveraging constraints of camera ego-motion and scene geometry. From a sequence of images, we first predict relative ego-motion between the input frames, and then reconstruct the camera trajectory. By considering the cycle consistency between 2D motion, depth and camera ego-motion, we train a model to reconstruct scene depth without additional supervision. These processes are trained using a self- supervised end-to-end convolutional neural network (CNN) architecture with motion field driven photometric consistency loss. To minimize accumulated error from imperfect local estimates, we predict relative reliability scores between every connected pair of input frames and then utilize them in global refinement.

In the second part, we reconstruct spatio-temporal 4D model from a set of 3D models. From a set of multiple 3D models, we optimize transformation parameters from each model space to global space using a Gaussian mixture to model point observations. We optimize alignment parameters using expectation-maximization algorithm and estimate the temporal extent for each 3D patches by maximizing expected posterior probability over time. This spatial-temporal model allows us to perform object segmentation as well as infer the existence of occluded objects.

Vision systems use a pipeline of feature extraction and analysis to predict the desired output from input image data. With robust features, we can achieve high accuracy using care- fully chosen, but simple analysis. Learning features and using features effectively are two problems we focus on in this work. We propose a novel formulation of convolutional sparse coding called spherical sparse coding (SSC). SSC removes the need for iterative optimization to compute the sparse codes (features) for the typical least squares formulation. Using the SSC formulation, we show a clear connection between convolutional sparse coding and con- volutional neural networks (CNNs). We extend SSC to a supervised method for classification that uses codes that are biased by a hypothesized class. These class-specific codes can be re- constructed to give us images that maximize the classification score of the hypothesized class. Next, we propose using the Siamense network architecture with the multi-channel normalized cross-correlation (MCNCC) similarity metric for cross-domain image matching. We show that our choices in how we use features can have significant performance consequences; even prior to learning any parameters, we achieve state-of-the-art performance using off-the-shelf CNN features with MCNCC on a number of different cross-domain matching problems.

One of the fundamental problems of computer vision is to detect and localize objects

such as humans and faces in images. Object detection is a building block for a wide

range of applications including self-driving cars, robotics and face recognition. Though

significant progress has been achieved in these tasks, it is still challenging to obtain

robust results in unconstrained images. Real world scenes usually contain more than one

object and it is very likely that some parts of an object are occluded by other objects

in the scene. To tackle occlusion, image features generated by occlusion should be explicitly

modeled rather than treated as noise. In this thesis, a deformable part model for detection

and keypoint localization is introduced that explicitly models part occlusion. The proposed

model structure makes it possible to augment positive training data with large numbers of

synthetically occluded instances. This allows us to easily incorporate the statistics of

occlusion patterns in a discriminatively trained model. To exploit bottom-up cues such as

occluding contours and image segments, we extend the proposed model to utilize bottom-up

class-specific segmentation in order to jointly detect and segment out the foreground

pixels belonging to the object.

In these approaches, a detector for a single object category is trained which operates

independently of other detections in the scene. An appealing alternative approach for

detection in cluttered images is to move from single object detection to whole-image

parsing. The presence of occlusion can then be “explained away” by the presence of an

occluding object. We model multi-object detection by classifying each pixel of the image

(semantic segmentation) using Convolutional Neural Network. CNN architectures have terrific

recognition performance but rely on spatial pooling which makes it difficult to adapt them

to tasks that require dense, pixel-accurate labeling. We demonstrate that while the

apparent spatial resolution of convolutional feature maps is low, the high-dimensional feature

representation contains significant sub-pixel localization information. We describe a

multi-resolution reconstruction architecture based on a Laplacian pyramid that uses skip

connections from higher resolution feature maps and multiplicative gating to successively

refine segment boundaries reconstructed from lower-resolution maps. We demonstrate that this approach yields state-of-the-art semantic segmentation results without resorting to more

complex random-field inference or instance detection driven architectures.

Humans are able to recognize objects in a scene almost effortlessly. Our visual system can easily handle ambiguous settings, like partial occlusions or large variations in viewpoint. One hypothesis that explains this ability is that we process the scene as a global instance. Using global contextual reasoning (e.g., a car sits on a road, but not on a building facade) can constrain interpretations of objects to plausible, coherent precepts. This type of reasoning has been explored in Computer Vision using weak 2D context, mostly extracted from monocular cues. In this thesis, we explore the benefits of strong 3D context extracted from multiple-view geometry. We demonstrate strong ties between geometric reasoning and object recognition, effectively bridging the gap between them to improve scene understanding.

In the first part of this thesis, we describe the basic principles of structure from motion, which provide strong and reliable geometric models that can be used for contextual scene understanding. We present a novel algorithm for camera localization that leverages search space partitioning to allow a more aggressive filtering of potential correspondences. We exploit image covisibility using a coarse-to-fine, prioritized search approach that can recognize scene landmarks rapidly. This system achieves state of the art results in large-scale camera localization, especially in difficult scenes with frequently repeated structures.

In the second part of this thesis, we study how to exploit these strong geometric models and localized cameras to improve recognition. We introduce an unsupervised training pipeline to generate scene-specific object detectors. These classifiers outperform state of the art and can be used when the rough camera location is known. When precise camera pose is available, we can inject additional geometric cues into novel re-scoring framework to further improve detection. We demonstrate the utility of background scene models for false positive pruning, akin to video-surveillance background subtraction strategies. Finally, we observe that the increasing availability of mapping data stored in Geographic Information Systems (GIS) provides strong geo-semantic information that can be used when cameras are located in world coordinates. We propose a novel contextual reasoning pipeline that uses lifted 2D

GIS models to quickly retrieve precise geo-semantic priors. We use these cues to to improve object detection and image semantic segmentation, providing a successful trade-off of false positives that boosts average precision over baseline detection models.

Two foundational and long-standing problems in computer vision are to detect and segment objects in images. The former problem focuses on finding certain classes of objects (e.g., humans), while the latter problem is concerned with obtaining a complete, pixelwise labeling of the scene, such that the objects of interest comprise different segments in the final labeling. Despite the obvious similarity between these two problems, they have typically been approached using techniques that have little in common.

In the first half of this thesis, we do away with the artificial distinction between segmentation and detection, and demonstrate the utility of producing segmentations from object detections directly in a feed-forward manner. Because a successful segmentation algorithm must respect bottom-up grouping cues, we begin by introducing a trainable edge detector that advances the state of the art in boundary detection, yet is extremely fast to train. Then, to inject top-down knowledge from object models into the segmentation process, we introduce a simple probabilistic model that captures the shape, appearance and depth ordering of a collection of detections in an image. As an application of this general idea, we release software that detects and segments cell nuclei in 3D confocal microscopy images, using detections as seeds to the segmentation pipeline.

When the underlying object detector is trained on a separate (often larger) dataset, there is usually room to tune it to perform better on new datasets. In the extreme, models sometimes do not generalize at all; this is often the case in applications to biology. To address this problem, we design our cell detector so that it can easily be retrained by an end user by labeling a few test images in order to adapt it to a new dataset. For modern techniques built to work on natural images, the usual way to handle dataset bias is to re-train on or fine tune to the new dataset, but both of these strategies require ground truth labels, which are laborious and often expensive to collect. Instead, we propose unsupervised methods for improving the quality of the underlying detector on new scenes given a large collection of photographs of the given scene, and show a substantial improvement over the baseline pre-trained models.

Figure-ground organization refers to the visual perception that a contour separating two regions belongs to one of the regions. Recent studies have found neural correlates of figure-ground assignment in V2 as early as 10-25 ms after response onset, providing strong support for the role of local bottom-up processing. How much information about figure-ground assignment is available from locally computed cues? Using a large collection of natural images, in which neighboring regions were assigned a figure-ground relation by human observers, we quantified the extent to which figural regions locally tend to be smaller, more convex, and lie below ground regions. Our results suggest that these Gestalt cues are ecologically valid, and we quantify their relative power. We have also developed a simple bottom-up computational model of figure-ground assignment that takes image contours as input. Using parameters fit to natural image statistics, the model is capable of matching human-level performance when scene context limited.