Automatic analysis of biological image datasets is one of the important achievements of applied image processing research. Feature extraction in volumetric bioimages obtained from numerous biomedical imaging techniques is becoming extremely critical for biologists and medical professionals to find the answers to many problems. In this work, we show how to extract effective features in two kinds of volumetric bioimages - plant shoot apical meristem(SAM) cell images taken by Confocal Laser Scanning Microscopy (CLSM) and the nematode images taken by Differential Interference Contrast Microscopy.

For actively developing tissues, a computational platform capable of automatically segmenting and tracking cells in the volumetric image stacks is very critical to obtaining high-throughput and quantitative spatiotemporal measurements of a range of cell behaviors, which will lead to a better understanding of the underlying dynamics of morphogenesis. The cells in the SAM are tightly clustered in space and have very similar shapes and intensity distributions, thus choosing reliable features to compute cell correspondences in space and time can be very challenging. In our research we propose a local graph matching based method to track the cells both spatially and temporally, and identify cell divisions. The geometric structure and topology of the cells' relative positions are efficiently exploited as the basic feature to match the cells. Furthermore, we build a joint segmentation and tracking system, where the tracking output acts as an indicator of the quality of segmentation and, in turn, the optimized segmentation can be improved to obtain better tracking results. In the end, the cell correspondences across multiple slices and time windows are fused together to obtain the final cell lineages. Experiments on multiple plant datasets show the proposed image analysis pipeline can effectively segment and track the SAMs cells.

Another contribution of this work on volumetric bioimages analysis lies in multilinear feature extraction and classification for nematode Digital Multi-focal Images (DMI). In such images, morphological information for a transparent specimen is captured in the form of a stack of high quality images, representing individual focal planes through the specimen's body. We present a method that can effectively exploit the entire information in the stack using the 3D X-Ray projections at different viewing angles. These DMI stacks represent the effect of different factors - shape, texture, viewpoint, different instances within the same class and different classes of specimens. For this purpose, we embed the 3D X-Ray Transform within a multilinear framework and propose a Multilinear X-Ray Transform feature representation. The experimental results on the nematode DMI data show that the proposed feature extraction and analysis method can give reliable recognition rate on a real-life database.

Analyzing scenes captured in video over wide areas, either in a single view or in multiple views, is the focus of this thesis. The thesis considers three inter-related problems in this domain - tracking in a network of cameras, summarization of the collected videos, and algorithm design that is robust to the physical conditions of the environment and the hardware platforms on which they are implemented. The inter-relationships between various objects in the scene (termed as context) are explored in the developed solution strategies.

In this thesis, we explore the problems of multi-target tracking in a wide-area scene, where the context information is shown to effectively improve the results because of the target interactions. We present a context-aware multi-target tracking algorithm in an overlapping camera network. The proposed algorithm is able to track multiple interacting targets in a wide-area camera network. From observations that both individuals and groups of targets interact with each other in natural scenes, we develop a Switching Network Tracker (SNT) that tracks both individuals and groups. In addition, due to the lack of the availability of public non-overlapping camera network datasets, we propose a new camera network tracking dataset (CamNeT) with multiple challenges. A baseline algorithm where the context information is considered is provided along with the dataset.

We also explore the video summarization problem. In a wide-area scene, most of the data is redundant. Therefore, we develop an algorithm, Context-Aware Video Summarization (CAVS) algorithm, that can capture the important video portions through information about individual local motion regions, as well as the interactions between these motion regions (context information). The summarization problem is then solved as the methodology of sparse coding with generalized sparse group lasso.

Moreover, although complex algorithms have been developed, there are few methods considering the real applications when there is a performance constraint. We study this fundamental problem in computer vision, which is to co-design the algorithm-platform for an unknown dataset under certain performance constraints. Our algorithm calculates a similarity between a test video and each unique training scenario. Similarity between training and test dataset indicates the same algorithm-platform can be applied to both of them. We test our algorithm on two applications: pedestrian detection and tracking.

We conclude the thesis by highlighting future research directions.

Images play an increasingly important role in many fields of science and its applications. Biology is one of the best examples of fields that have come to depend heavily upon images for progress. Biological images contain a lot of objects and patterns, which may contain information about underlying mechanism in biology. Image analysis provides a way to extract and quantify objects and patterns in image data and obtain answers to meaningful biological questions. This dissertation focuses on image analysis for biological applications in segmentation, registration and count estimation problems.

Proper understanding of the causal relationship between cell growth patterns and gene expression dynamics is one of the major topics of interest in developmental biology. Information such as rates and patterns of cell expansion play a critical role in explaining cell growth and deformation dynamics. In our research, we focus on studying the developing plant with the goal of obtaining very accurate cell development statistics. The image processing and analysis framework for gathering the cell growth and division statistics comprises of three main parts - image registration, cell segmentation and cell tracking. Without proper segmentation and registration the subsequent parts in the image analysis system would fail. This dissertation addresses both these problems.

To provide proper segmentation of cells we propose a single framework that entails segmentation and tracking of plant cell images. We show how to optimally choose the parameters in the watershed segmentation algorithm for high quality segmentation results. To register image datasets we have provided an optimization based framework to select the best image slice correspondence from consecutive image stacks by using the tissue characteristics in images. Also, we have presented a novel landmark selection method where we use characteristics of neighboring cells as unique features. To evaluate both our frameworks, cell correspondences across multiple slices and

time windows are fused to obtain cell lineages and compared to recent results in this area. Experiments on multiple plant datasets show the proposed algorithms provide significantly longer, more accurate cell lineages and more comprehensive identification of cell divisions.

Another contribution of this work is in count estimation. Mosquitoes

and other blood-feeding insects transmit deadly diseases to hundreds of millions of people, causing severe suffering and more than a million deaths each year. Female mosquitoes that transmit deadly diseases locate human hosts by detecting exhaled CO2 and skin odor. Quantitative analysis of a mosquito’s attraction to different odors is very important. It can lead to discoveries about mosquitoes that could have a big impact on human health. We present a method to automate mosquito counting in videos. The current manual analysis on these videos is not sufficient for quantitative analysis and available object counting approaches do not work well. We propose an automated mosquito counting technique which uses the softmax classifier with a sliding window

technique to detect the mosquitoes in the separated region of interest, where the over-detections resulting from the sliding window technique are eliminated by non-maximum suppression method. The final counts of the mosquito detections represent the desired mosquito counts in images. The proposed automated method has been applied on different datasets and showed very good and consistent counts of mosquitoes.

Egocentric vision captures the scene from the point of the view of the camera wearer while exocentric vision captures the overall scene context. Jointly modelling ego and exo views is a crucial step towards developing next-generation AI agents. The community has regained interest in the field of egocentric vision. While, third-person view and first-person has been thoroughly investigated, very few works aim to study the both synchronously. Exocentric videos contain many relevant signals transferrable to egocentric videos. We propose a multimodal-LLM model that leverages large-scale exocentric information for the task of egocentric action recognition. This thesis also provides a broad overview of works combining both the egocentric and exocentric vision.

With the increasing popularity of mobile cameras in various computer vision and multimedia applications, the demand for high-quality visual content is also increasing. However, videos captured using current consumer-grade cameras often suffer from a variety of quality issues such as motion blur, low frame-rate, low resolution, and rolling shutter artifacts. The reasons vary, including low shutter frequency, long exposure times, type of imaging sensors, and the movement of the device itself. These factors limit the quality of videos captured. As a vast majority of videos is captured using mobile cameras these days, it calls for improved quality of the video captured by these devices. In this thesis, we focus on enhancing the quality of videos by leveraging the spatio-temporal internal structure of the given video along with the external information available from the external dataset. Most of the existing works make a prior assumption on the degradation model that affects the quality of videos. Examples of such assumptions in degradation models include knowledge that all input frames are blurry, known degradation kernel for spatio-temporal down-sampling, and absence of rolling shutter artifacts. Nevertheless, in many real-world applications, these assumptions don't hold true as the input priors are usually unknown. We term these unknown priors as blind priors for the task of video enhancement. In this regard, we first present our work on joint video deblurring and interpolation with no prior assumption that input frames are always blurry. We utilize internal information available from neighbouring frames to deblur and interpolate between frames. Then, we describe our approach on blind spatio-temporal video super-resolution with the unknown down-sampling kernel, by leveraging an external dataset and internal structure of a given video. Next, we present our work on joint rolling shutter correction and super-resolution to recover the high-resolution global shutter video using patch-recurrence property in videos. Finally, we show an application of enhancement techniques in biomedical imaging, where we utilize quantization in feature space for unsupervised image denoising. We demonstrate that the proposed approaches effectively utilize the internal learning for the task of video enhancement and show impressive performance in different real-world blind prior settings.

As networks of video cameras are being installed in many applications, modeling and inference strategies in video networks have captured more and more interest. There are many challenge problems, such as (i) traditional computer vision challenges in tracking and recognition, robustness to pose, illumination, occlusion, clutter; recognition of objects and activities; (ii) aggregating local information to obtain stable, long-term tracks of objects; (iii) cooperative camera control algorithms for multi-resolution target acquisition; (iv) distributed processing and scene analysis and (v) communication in a distributed manner. The overall aim of this thesis is to study the core issues in network-centric processing, control and communication in a multi-camera network, including frameworks for tracking people in video through changes of activities, tracking in a non-overlapping camera network, decentralized control and tracking in a camera network, and distributed video compression.

In our work, we address the problem of tracking in camera networks by dividing into two parts: tracking people through changes in activities and tracking multiple targets in a network of non-overlapping cameras. We present a novel framework for tracking a long sequence of human activities, including the time instances of change from one activity to the next, using a non-linear switching dynamical feedback system. Meanwhile, a multi-objective optimization framework by combining short term feature correspondences across the cameras with long-term feature dependency models is proposed to take care of tracking in a non-overlapping network.

We also deal with the problem of decentralized, cooperative control of a camera network and distributed multi-target tracking in such a network of self-configuring pan-tilt-zoom cameras. This control cannot be based on separate analysis of the sensed video in each camera. They must act collaboratively to be able to acquire multiple targets at different resolutions. Our research focuses on developing accurate and efficient camera control algorithms in such scenarios using game theory. For tracking the targets as they move through the area covered by the cameras, we propose a special application of the distributed estimation algorithm known as Kalman-Consensus filter through which each camera comes to a consensus with its neighboring cameras about the actual state of the target.

Finally, we present a framework for multi-terminal video compression (MTVC) that exploits the geometric constraints between cameras with overlapping fields of view, and then uses distributed source coding on corresponding points in two or more views. Our proposed method is composed of two parts - a Distributed Motion Estimation (DME) algorithm and a Distributed Source Coding (DSC) algorithm.

Due to their high fault-tolerance, ease of installation and scalability to large networks, distributed algorithms have recently gained immense popularity in the sensor networks community, especially in computer vision. Multi-target tracking in a camera network is one of the fundamental problems in this domain. Distributed estimation algorithms work by exchanging information between sensors that are communication neighbors. Since most cameras are directional sensors, it is often the case that neighboring sensors may not be sensing the same target. Such sensors that do not have information about a target are termed as ``naive'' with respect to that target. State-of-the-art distributed state estimation algorithms (e.g., the Kalman Consensus Filter (KCF)) in the sensor networks community are not directly applicable to tracking applications in camera networks mainly due to this naivety issue.

In our work, we propose generalized distributed algorithms for state estimation in a sensor network taking the naivety issue into account.

For multi-target tracking, along with the tracking framework, a data association step is necessary where the measurements in each camera's view are associated with the appropriate targets' tracks. At first, under the assumption that the data association is given, we develop distributed state estimation algorithms addressing the naivety issue. In this process, first, we propose the Generalized Kalman Consensus Filter (GKCF) where an information-weighting scheme is utilized to account for the naivety issue. Next, we propose the Information-weighted Consensus Filter (ICF) which can achieve optimal centralized performance while also accounting for naivety. This is the core contribution of this thesis.

Next, we introduce the aspect of multi-target tracking where a probabilistic data association scheme is incorporated in the distributed tracking scheme resulting the Multi-Target Information Consensus (MTIC) algorithm. The incorporation of the probabilistic data association mechanism makes the MTIC algorithm very robust to false measurements/clutter.

The aforementioned algorithms are derived under the assumption that the measurements are related to the state variables using a linear relationship. However, in general, this is not true for many sensors including camera sensors. Thus, to account for the non-linearity in the observation model, we propose non-linear extensions of the previous algorithms which we denote as the Extended ICF (EICF) and the Extended MTIC (EMTIC) algorithms. In-depth theoretical and experimental analysis are provided to compare these algorithms with existing ones.

In most domain adaptation methods, concurrent access to source and target data isneeded. In many real-life problem statements, it’s highly likely that the access to the source data might not be possible during the adaptation process. Source-free domain adaptation methods, while very necessary, usually have significant performance gap when compared with those that assume access to the source data. To tackle this, we introduce a source free domain adaptation strategy for the semantic segmentation problem that internally uses active learning on the pixel level predictions (i.e., pseudo labels) to obtain a framework that can generalize well between the source and the target domain. Given a small budget for labeling, we select the pixels needed to be sent to an oracle for labeling. These labeled pixels are then used to boost performance of the the overall system. Our method does not assume any source data or any prior labels available on the target data. We observe that even without the source data and only by labeling a small percentage of pixels using active learning, we can still reach comparable performance with many standard unsupervised domain adaptive semantic segmentation methods.We also show a brief example application of domain adaptation on the environmental scenario

We present a multi-disciplinary framework for motion modeling and recognition in machine vision. Building upon the neurobiological model of motion recognition, we propose computational equivalents for the Motion Energy and Form Pathways. We derive the Hamiltonian Energy Signature (HES) from first-principles in physics as the basis for the Motion Energy Pathway. The Form Pathway is modeled using existing low-level feature descriptors based on shape, appearance, and gradients. We propose an Integration methodology to combine both pathways using a variant of the Feature Integration and Biased Competition neurobiological models, which we implement via statistical hypothesis testing using the bootstrap. We also show the extensibility of our physics-based approach by proposing a physically-significant, compact representation for the gait of a person called the Gait Action Image (GAI), which is based on core physics principles employed in the HES formulation. We then show the generalizability of our neurobiologically-inspired integration framework by casting the GAI within this infrastructure.

Since Motion and image analysis are both important for activity recognition in video, we also develop a new approach that extends the Hamiltonian Monte Carlo (HMC) to allow us to simultaneously search over the combined motion and image space in a concerted manner using well-known Markov Chain Monte Carlo (MCMC) techniques. For motion analysis in video, we use tracks generated from the video to calculate the Hamiltonian equations of motion for the systems under study, thus utilizing analytical Hamiltonian dynamics to derive a physically significant HMC algorithm which can be used for activity analysis. We then use image analysis to help explore both the motion energy space and the image space by integrating the Hamiltonian energy-based approach with an image-based data-driven proposal to drive the HMC, thereby yielding a Data Driven HMC (DDHMC). We reduce the enormity of the search space by driving the Hamiltonian dynamics-based MCMC with image data in this DDHMC. We also develop the reverse algorithm, which uses motion energy proposals to search the image space. While HMC has been used in other contexts, this is possibly the first work that shows how it can be used for activity recognition in video, taking into account the image analysis results and using the physical motion information of the system. In addition, the DDHMC framework has potential application to other domains where statistical sampling techniques are useful, as we outline in the section on future work.

Experimental validation of the theory is provided on the well-known KTH, Weizmann, and USF Gait datasets with very promising results.