Human are interpolating the visual world with very rich understanding. For example, when observing the world through eyes, we not only understand the high level semantic meaning of each region/pixel, more importantly, we also understand the 3D properties like how far away each object is and how the 3D shape of each object is in order to do interaction with the world. In the field of computer vision, however, visual understanding are separated into multiple tasks, e.g. segmentation, 3D reconstruction or object detection etc., due to its high complexity. However, this induces the problem that the results from different strategies are lack of compatibility among different tasks. For example, semantic object detection can not take care of the 3D occlusion regions, while 3D reconstruction does not consider overall semantic context. Thus, in order to have good visual understanding, it is critical to joint understand different tasks while maintaining their compatibility.

Luckily, thanks to the raising technique of deep learning, (a.k.a. convolutional neural network (CNN)), which dramatically beats the other traditional strategies in many visual tasks based on hierarchical learned features with a nearly single framework, we are able to unify different understandings in a more compact and efficient way by designing reasonable output and interaction terms.

However, CNN is not a magic key of solving all problems, and one obvious limitation of CNN is that it contains arbitrarily selected convolutional kernel size and layers, yielding non-adaptive receptive fields to match the variance of object scales. In addition, it is not strait-forward to add arbitrary connections inside each layer based on intuition. Thus, we further embed the conditional random field (CRF) into the system in order to compensate the deficiency in order to unify different cues and perform multiple tasks simultaneously.

In this thesis, we prove the concept through estimating multiple tasks jointly including joint part and object segmentation, joint segmentation and geometry estimation etc. We first show that we can fit deep convolutional network into many different tasks to acquire superior performance compare to traditional shallow features. Secondly, by unifying different tasks with our designed compatibility constrains, we make different tasks mutually regularized and beneficial. Finally, to evaluate the results, we perform our experiments over the standard evaluating benchmarks like PASCAL for segmentation and the NYU v2 dataset for depth estimation. Last but not the least, we not only apply the existing metrics to show the performance gain from our design, but also introduce reasonable new metrics in order to better show the aspect that improved.

We are interested in training complementary networks for large-scale image classification. This is motivated by the observation that deep neural networks learn different visual concepts from the foreground and background regions (defined by the object bounding box) of natural images. We construct complementary training image sets for this purpose. That is to say, in each training image, either the foreground or background region, defined by object bounding boxes, is removed. We train deep neural networks on these modified datasets, and show the possibility of image classification even using a network trained on pure background visual contents. We visualize the neural networks, and demonstrate that networks trained with different datasets capture complementary information. These complementary networks are combined at the testing stage on two conditions with and without bounding box(es), producing remarkable gain in recognition accuracy.

In this paper, we propose an approach to classify action sequences. We observe that in action sequences the critical features for discriminating between actions occur only within sub-regions of the image. Hence deep network approaches will address the entire image are at a disadvantage. This motivates our strategy which uses static and spatio-temporal visual cues to isolate static and spatio-temporal regions of interest (ROIs). We then use weakly supervised learning to train deep network classifiers using the ROIs as input. More specifically, we combine multiple instance learning (MIL) with convolutional neural networks (CNNs) to select discriminative action cues. This yields classifiers for static images, using the static ROIs, as well as classifiers for short image sequences (16 frames), using spatio-temporal ROIs. Extensive experiments performed on the UCF101 and HMDB51 benchmarks show that both these types of classifiers perform well individually and achieve state of the art performance when combined together. We also show qualitatively that our ROIs (selected by the algorithms) capture the most relevant parts of the image sequences.

Modeling object is one of the core problems in computer vision. A good object model can be applied to multiple visual recognition tasks. In this thesis, we use compositional relations to model the objects and parts for articulated objects, which is very challenging due to the large variability by subtypes, viewpoints and poses. Intuitively, objects are composed of parts, and each part is then composed of subparts. By recursively applying the compositional relations, we can obtain a hierarchical structure of the object, which is called compositional model. Compared to the popular black-box deep learning system, our model enjoys the advantages of explainability, and adaptivity to more complex scenarios. There are three challenges for the compositional model: 1) How to learn the parts and subparts, 2) How to learn the compositional structure between parts and subparts, 3) How to allow efficient learning and inference. In this thesis, we mainly focus on three relevant projects using the concept of compositions and parts.

In the first project, we study the problem of semantic part segmentation for animals. Compositional model is used to represent boundaries of objects and parts. Given part-level boundary annotations, a mixture of models is learned to deal with various viewpoints and poses. We also incorporate edge, appearance, and semantic part cues into the compositional model. Besides, a linear complexity algorithm is developed for efficient inference. The evaluation using horse and cow images from PASCAL VOC demonstrates the effectiveness of our method.

In the second project, we provide a novel unsupervised method to learn object semantic parts from internal states of CNNs. Our hypothesis is that semantic parts are represented by populations of neurons rather than by single filters. We propose a simple clustering technique to extract part representations, which we call visual concepts. We qualitatively show that visual concepts are semantically coherent in that they represent most semantic parts of the object, and visually coherent in that their corresponding image patches appear very similar. Next, we treat the visual concepts as part detectors and quantitatively evaluate their performance for detecting semantic parts. The experiments on the PASCAL3D+ dataset and our newly annotated VehicleSemanticPart dataset support that CNNs have internal representations of object parts encoded by populations of neurons.

In the third project, we use the visual concepts for detecting the semantic parts on partially occluded objects. We consider a scenario where the model is trained using non-occluded images but tested on occluded images. The motivation is that there are infinite number of occlusion patterns in real world, so the models should be inherently robust and adaptive to occlusions instead of learning the occlusion patterns in the training data. Our approach uses a simple voting scheme, based on log-likelihood ratio tests and spatial constraints, to combine the evidence of local cues, which are derived from visual concepts. Experiments show that our algorithm outperforms several competitors in semantic part detection, especially in the presence of occlusions.