This thesis presents three principled approaches to dense pixel level labeling and demonstrates their effectiveness for both segmentation and boundary detection.

First, a structured decision tree classifier for vessel wall segmentation on volumetric angiograms is presented. Building upon advances in natural image boundary detection, this 3D classifier generated structured patch-to-patch 3D vessel boundary localization using domain specific volumetric features, an adaptive prior coupled with an importance driven sampling scheme. Through comparison of our methodologies to a number of baselines including widely used edge detection strategies, non-structured decision trees, and approaches using alternative input features the effectiveness of our classifier is demonstrated. Additionally, the classifier is shown to be robust to error in the a-priori information.

Second, we describe a 3D Convolutional Neural Network (CNN) approach to pixel classification. Two CNN classifiers are presented, HED-3D and I2I. The first extends the popular Holistically-Nested Edge Detector into 3D to perform generic volumetric segmen- tation. A second 3D CNN is introduced that performs precise localization using a novel fine-to-fine, multi-scale architecture. This classifier addresses three key issues to precise image-to-image and volume-to-volume labeling: 1) efficient end-to-end voxel label prediction and training, 2) precise localization capable of capturing fine structures typical in medical data, and 3) direct multi-scale, multi-level representation learning. Our 3D CNN is shown to out-perform alternative fine-to-fine strategies through demonstration and evaluation on multiple data-sets and tasks. We evaluate these frameworks on three challenging tasks, vessel boundary prediction, brain boundary prediction and skull-stripping.

Lastly, our fine-scale localization method is augmented with spatial context process- ing to perform automatic 3D cardiovascular model construction from medical image data. This approach builds upon the I2I architecture to generate accurate segmentation as part of DeepLofting, an efficient pipeline for 3D cardiovascular model construction. The I2I classifier is extended to use spatial context during prediction, forming a new classifier I2I-FC. This powerful classifier is a critical component in DeepLofting, which builds 3D cardiovascular models from medical volumes and a-priori information. DeepLofting is evaluated on a publicly available cardiovascular model dataset, and represents a critical step forward in 3D model generation.