Technological advances in digital imaging, broadband networking, and data storage are motivating millions of people to communicate with one another and express themselves by sharing images, video, and other forms of media online. Acquiring, storing, and transmitting photos is now trivial, but to manipulate, index, sort, filter, summarize, or search through them is significantly harder. The Statistical Visual Computing Laboratory at the University of California, San Diego, has been considering the problem of content-based image retrieval for years. This effort explores many issues in image representation and intelligent system design, including the evaluation of image similarity, automatic annotation of images with descriptive captions, and the ability to understand user feedback during image search.

Large-scale datasets play a key role in the success of modern computer vision models. However, there are limitations of current datasets due to the data collection strategy. When an object has a canonical pose, pose bias is usually common. And models can only perform good on specific poses. We mitigate this by introducing two data augmentation methods, one in feature space with feature transfer, the other in image space with novel view synthesis. Both of them can improve the pose diversity of the current dataset. Imbalance class distribution is the nature of real world. To deal with this problem, we first discuss the few-shot open-set problem. By introducing meta-learning to open-set detection, PEELER rejects unseen samples with no cost of seen class accuracy. With a larger class label space, long-tailed recognition problem is then discussed. GistNet improve the long-tailed performance by geometry transfer, while Breadcrumb mitigates the over-fitting of few-shot samples by feature back-tracking. Last but not least, the task of semi-supervised long-tailed recognition is introduced. Alternate learning is presented to combine semi-supervised learning with long-tailed recognition. All of above mentioned methods are proved to be useful when facing certain limitations of computer vision datasets.

Object detection is one of the most important and challenging research topics in computer vision. It is playing an important role in our everyday life and has many applications, e.g. surveillance, autonomous driving, robotics, drone, medical imaging, etc. The ultimate goal of object detection is a universal object detector that can work very well in any case under any condition like human vision system. However, there are multiple challenges on the universality of object detection, e.g. scale-variance, high-quality requirement, domain shift, computational constraint, etc. These will prevent the object detector from being widely used for various scales of objects, critical applications requiring extremely accurate localization, scenarios with changing domain priors, and diverse hardware settings. To address these challenges, multiple solutions have been proposed in this thesis. These include an efficient multi-scale architecture to achieve scale-invariant detection, a robust multi-stage framework effective for high-quality requirement, a cross-domain solution to extend the universality over various domains, and a design of complexity-aware cascades and a novel low-precision network to enhance the universality under different computational constraints. All these efforts have substantially improved the universality of object detection, and the advanced object detector can be applied to broader environments.

While CNNs have enabled tremendous progress in computer vision for a variety of tasks, robust generalization across domain, viewpoint, and class remains a significant challenge. This thesis therefore centers around a new hierarchical multiview, multidomain image dataset with 3D meshes called 3D-ODDS. Data was collected in several ways, involving turntable setups, flying drones, and in-the-wild photos under diverse indoor/outdoor locations. Experiments were subsequently conducted on two important vision tasks: single view 3D reconstruction and image classification. For single view 3D reconstruction, a novel postprocessing step involving test-time self-supervised learning is proposed to help improve reconstructed shape robustness. For image classification, we consider an adversarial attack framework using perturbations which are semantically imperceptible based on human subject surveys. For both tasks, experiments show that 3D-ODDS is a challenging dataset for state of the art methods and is useful in measuring class, pose, and domain invariance. We believe that the dataset will remain relevant moving forward, inspiring future works towards robust and invariant methods in computer vision.

Visual recognition is a problem of significant interest in computer vision. The current solution to this problem involves training a very deep neural network using a dataset with millions of images. Despite the recent success of this approach on classical problems like object recognition, it seems impractical to train a large scale neural network for every new vision task. Collecting and correctly labeling a large amount of images is a big project in itself. The process of training a deep network is also fraught with excessive trial and error and may require many weeks with relatively modest hardware infrastructure. Alternatively one could leverage the information already stored in a trained network for several other visual tasks using transfer learning.

In this work we consider two novel scenarios of visual learning where knowledge transfer is affected from off-the-shelf convolutional neural networks (CNNs). In the first case we propose a holistic scene representation derived with the help of pre-trained object recognition neural nets. The object CNNs are used to generate a bag of semantics (BoS) description of a scene, which accurately identifies object occurrences~(semantics) in image regions. The BoS of an image is, then, summarized into a fixed length vector with the help of the sophisticated Fisher vector embedding from the classical vision literature. The high selectivity of object CNNs and the natural invariance of their semantic scores facilitate the transfer of knowledge for holitistic scene level reasoning. Embedding the CNN semantics, however, is shown to be a difficult problem. Semantics are probability multinomials that reside in a highly non-Euclidean simplex. The difficulty of modeling in this space is shown to be a bottle-neck to implementing a discriminative Fisher vector embedding. This problem is overcome by reversing the probability mapping of CNNs with a natural parameter transformation. In the natural parameter space, the object CNN semantics are efficiently combined with a Fisher vector embedding and used for scene level inference. The resulting semantic Fisher vector achieves state-of-the-art scene classification indicating the benefits of BoS based object-to-scene transfer.

To improve the efficacy of object-to-scene transfer, we propose an extension of the Fisher vector embedding. Traditionally, this is implemented as a natural gradient of Gaussian mixture models (GMMs) with diagonal covariance. A significant amount of information is lost due to the inability of these models to capture covariance information. A mixture of Factor analyzers (MFAs) are used instead to allow efficient modeling of a potentially non-linear data distribution in the semantic manifold. The Fisher vectors derived using MFAs are shown to improve substantially over the GMM based embedding of object CNN semantics. The improved transfer-based semantic Fisher vectors are shown to outperform even the CNNs trained on large scale scene datasets.

Next we consider a special case of transfer learning, known as few-shot learning, where the training images available for the new task are very few in number (typically less than 10). Extreme scarcity of data points prevents learning a generalize-able model even in the rich feature space of pre-trained CNNs. We present a novel approach of attribute guided data augmentation to solve this problem. Using an auxiliary dataset of object images labeled with 3D depth and pose, we learn trajectories of variations along these attributes. To the training examples in a few-shot dataset, we transfer these learned attribute trajectories and generate synthetic data points. Along with the original few-shot examples, the additional synthesized data can also be used for the target task. The proposed guided data augmentation strategy is shown to improve both few-shot object recognition and scene recognition performance.

The success of deep learning in image recognition is substantially driven by large-scale, well-curated data. On visual recognition of common objects, the data can be scalably annotated on online crowd-sourcing platforms because the labeling does not need any prior knowledge. However, the case is not true for images of expertise like biological or medical imaging in which labeling them needs background knowledge. Although data collection is still usually easy, the annotation is difficult. Existing self-supervised or semi-supervised solutions train a model that tries to learn from a small amount of labeled data and a large amount of unlabeled data. These solutions show good performances on common object recognition but have been found not to work effectively on fine-grained expert domains.

In this thesis, we propose a new solution with crowd source annotations to address the problem. Inspired by the fact that supervised learning on as much as data can always perform better, our method tries to scale up the annotation. This is implemented by two different approaches, machine teaching and human filtering. Machine teaching first teaches humans with a short carefully designed course to learn the expertise knowledge so that they can label the data later. Human filtering simplifies the process to a binary selection procedure without preceding training. Beyond these two approaches, a unified explanation framework is developed to generate visualizations that are merged into two approaches, enabling easier and more accurate annotation results. Experiments show that both methods significantly outperform various alternative approaches in several benchmarks. They have also been found to be versatile and can benefit from more advanced machine learning techniques in the future. Overall, we believe that this thesis opens up a new direction to think about the expert domain classification problem, in general.

Deep learning has brought remarkable improvement for the performance of image recognition tasks. However, the resource limitation forms a big obstacle for the real application of deep learning. Two types of resource constraints- limited machine's computing power and lack of annotated data are considered in the thesis. Compared to leveraging a large static network formed by input independent blocks, we try to overcome the issues of resource limitation with a more effective architecture by proposing a series of dynamic neural networks with input dependent blocks.

For the tasks with constrained computational resources, we first consider the multi-domain learning problem, which requires a single framework to perform well on multiple datasets for image classification. CovNorm is proposed to dynamically project a common feature to different feature spaces according to the dataset ID by consuming tiny numbers of extra parameters and computation. Then a large scale image recognition problem under different computational resources is explored. Dynamic Convolution Decomposition (DCD) is proposed for the machines with computing power from the order of 100 MFLOPs to 10 GFLOPs while MicroNet is designed to be applied on the machines with a computational cost far below 100 MFLOPs. Empowered by the dynamic architecture, both DCD and MicroNet achieve a significant improvement within their working scope.

For the issue of lacking annotated data, we work on the domain adaptation tasks, where the dataset is partially labeled and a domain gap exists between the labeled data (source domain) and the unlabeled data (target domain). We start by considering a relatively simple case with a single source and single target domain on semantic segmentation. A bidirectional learning (BDL) framework is designed and it reveals the synergy of several key factors, i.e., adversarial learning and self-training for domain adaptation. Based on the techniques given by BDL and the power of dynamic networks, a more complex problem- multi-source domain adaptation is investigated. Dynamic residual transfer (DRT) is presented and shows tremendous improvement for the adaptation performance compared to its static version. It confirms the effectiveness of dynamic networks for the image recognition problem when the amount of annotated data is limited.