UC San Diego

This dissertation introduces the concept of content- oriented 3D reconstruction from image streams. The presented research enables reconstruction algorithms to use content from different stages in the reconstruction pipeline to focus given resources to areas of interest. The content used can come from previous reconstruction results or input images and leads to focused reconstructions, region-of-interest refinements, regions of error extractions or input data filtering. In general content is used to improve reconstruction results by either focusing resources on target objects or filtering input or output data that reduces the quality of the reconstruction. Several case studies for content-oriented 3D reconstructions for two different reconstruction environments are introduced. These two environments consist of a real-time reconstruction used for tele- immersion and structure and motion used for reconstructions from aerial imagery. Information from the present frame and also from previous frames is used to focus the efforts on the reconstruction targets in the case of real-time reconstruction. This focusing is performed in two case studies including real-time stereo as well as volumetric visual hull reconstructions. In the case of the volumetric visual hull, a region-of-interest refinement using content from the initial reconstruction of each frame is introduced to further improve visual results. These approaches lead to high resolution avatar reconstructions at interactive rates. The developed techniques and algorithms were translated into a proof-of- concept system, allowing users to interact with remote collaborators and virtual worlds. Futhermore, content oriented error extraction and factorization for unconstraint dense correspondences and cameras is introduced in the structure and motion environment. This enables either filtering of erroneous results or localized improvements of the reconstruction. A frame decimation approach is introduced that filters the input streams based on their content to avoid ill-posed input data. Both approaches help to reduce errors in the used multi-view reconstructions pipeline. The presented research illustrates that content-based 3D reconstructions from image streams result in higher quality structure extraction while reducing the computational complexity, resulting in increased reconstruction speeds