This dissertation introduces the concept of content- oriented 3D reconstruction from image streams. The presented research enables reconstruction algorithms to use content from different stages in the reconstruction pipeline to focus given resources to areas of interest. The content used can come from previous reconstruction results or input images and leads to focused reconstructions, region-of-interest refinements, regions of error extractions or input data filtering. In general content is used to improve reconstruction results by either focusing resources on target objects or filtering input or output data that reduces the quality of the reconstruction. Several case studies for content-oriented 3D reconstructions for two different reconstruction environments are introduced. These two environments consist of a real-time reconstruction used for tele- immersion and structure and motion used for reconstructions from aerial imagery. Information from the present frame and also from previous frames is used to focus the efforts on the reconstruction targets in the case of real-time reconstruction. This focusing is performed in two case studies including real-time stereo as well as volumetric visual hull reconstructions. In the case of the volumetric visual hull, a region-of-interest refinement using content from the initial reconstruction of each frame is introduced to further improve visual results. These approaches lead to high resolution avatar reconstructions at interactive rates. The developed techniques and algorithms were translated into a proof-of- concept system, allowing users to interact with remote collaborators and virtual worlds. Futhermore, content oriented error extraction and factorization for unconstraint dense correspondences and cameras is introduced in the structure and motion environment. This enables either filtering of erroneous results or localized improvements of the reconstruction. A frame decimation approach is introduced that filters the input streams based on their content to avoid ill-posed input data. Both approaches help to reduce errors in the used multi-view reconstructions pipeline. The presented research illustrates that content-based 3D reconstructions from image streams result in higher quality structure extraction while reducing the computational complexity, resulting in increased reconstruction speeds

A correspondence and camera error analysis for dense correspondence applications such as structure from motion is introduced. This provides error introspection, opening up the possibility of adaptively and progressively applying more expensive correspondence and camera parameter estimation methods to reduce these errors. The presented algorithm evaluates the given correspondences and camera parameters based on an error generated through simple triangulation. This triangulation is based on the given dense, non-epipolar constraint, correspondences and estimated camera parameters. This provides an error map without requiring any information about the perfect solution or making assumptions about the scene. The resulting error is a combination of correspondence and camera parameter errors. An simple, fast low/high pass filter error factorization is introduced, allowing for the separation of correspondence error and camera error. Further analysis of the resulting error maps is applied to allow efficient iterative improvement of correspondences and cameras.

This paper introduces a non-parametric sequential frame decimation algorithm for image sequences in low-memory streaming environments. Frame decimation reduces the number of input frames to increase pose and structure robustness in Structure and Motion (SaM) applications. The main contribution of this paper is the introduction of a sequential low-memory work-flow for frame decimation in embedded systems where memory and memory traffic come at a premium. This approach acts as an online preprocessing filter by removing frames that are ill-posed for reconstruction before streaming. The introduced sequential approach reduces the number of needed frames in memory to three in contrast to global frame decimation approaches that use at least ten frames in memory and is therefore suitable for low-memory streaming environments. This is moreover important in emerging systems with large format cameras which acquire data over several hours and therefore render global approaches impossible. In this paper a new decimation metric is designed which facilitates sequential keyframe extraction fit for reconstruction purposes, based on factors such as a correspondence-to-feature ratio and residual error relationships between epipolar geometry and homography estimation. The specific design of the error metric allows a local sequential decimation metric evaluation and can therefore be used on the fly. The approach has been tested with various types of input sequences and results in reliable low-memory frame decimation robust to different frame sampling frequencies and independent of any thresholds, scene assumptions or global frame analysis.

An algorithm that shows how ray divergence in multi-view stereo scene reconstruction can be used towards improving bundle adjustment weighting and conditioning is presented. Starting with a set of feature tracks, ray divergence when attempting to compute scene structure for each track is first obtained. Assuming accurate feature matching, ray divergence reveals mainly camera parameter estimation inaccuracies. Due to its smooth variation across neighboring feature tracks, from its histogram a set of weights can be computed that can be used in bundle adjustment to improve its convergence properties. It is proven that this novel weighting scheme results in lower reprojection errors and faster processing times than others such as image feature covariances, making it very suitable in general for applications involving multi-view pose and structure estimation.