Time lapse 3D images are important resources for biology research because they notonly provide 3D structural information but also provide temporal information. Much of the analysis of these data are manual and subjective, and does not scale well with the large amount of imaging data that is routinely collected these days. This is especially true for 3D and time-lapse imagery. Fundamental problems such as detecting cells, subcellular features, tracking of such structures in 3D over time, and modeling 3D shapes in robust manner, remain. The problems addressed in this dissertation are motivated by two bio-imaging problems:

1. Understanding the plant pavement cell growth pattern. The pavement cell growthcontrols the leaf expansion patterns and rates which are the key determinants of the overall photosynthetic rates of the canopy. Time lapse image stacks from 3D confocal imagery are a good resource to study the pavement cell growth process. To better understand this process, machine learning methods are developed to detect and track cellular and sub-cellular features. We developed a deep learning enabled time-lapse 3D analysis pipeline that includes novel boundary tagged 3D segmentation method, and a graph based sub-cellular feature extraction and tracking method. Detailed quantitative evaluation results demonstrate the robustness and state-of-the art performance of the proposed methods.

2. Neuron morphology analysis. Cell morphology especially neuron morphology playsan important role in biology because neuron functions are closely related to neuron morphology. In this dissertation, we propose a robust computational 3D skeleton model to analyze neuron morphology. It is the first deep learning method to compute a 3D neuron skeleton model directly from discrete 3D surface points for neuron classification. The main innovation is in formulating the learning problem associated with computing the medial axis transform that represents the 3D skeleton. We apply our method on two Datasets, Ciona neuron dataset and C.elegan neuron dataset for classification. It results in an accurate and robust skeleton representation, and achieves state-of-the-art performance in classifying neuron types.

The implementation of the methods developed above and the associated data aremade available on GitHub and also as a software service through the UCSB BisQue platform.

Studying spatial arrangement and relationships in full tissue samples can improve our understanding of the various developmental/pathological processes that underlie proper organ or organism function. In particular, it has been found that neuronal or vascular structures are pervasive in many tissues, and oftentimes are spatially correlated with other cells. This work aims to discover those relationships, by extracting biological knowledge from cellular and sub-

cellular imaging using spatial point process methods.

In this dissertation, we present discoveries on spatial distributions and attributes of dendritic spines and retinal astrocytes, two crucial elements in the mammalian nervous system. Although little is known about the spatial distributions of either respective to their surroundings and attributes, this thesis attempts to pose some possible biological hypotheses based on strong statistical evidence, as well as further extend the tools used for spatial analysis. In particular, we develop a multitype version of the linear network K-function, a summary function used for measuring clustering or repulsion of point features existing on a linear network.

Numerous large vision models for natural images, such as SAM, Florence-2, and GPT-4, have achieved state-of-the-art (SOTA) performance, largely due to vast amounts of image and text data available online. Smaller models like EfficientSAM and CLIP have also shown the potential of achieving significant results with comparatively less data. However, real-world scientific problems, particularly in remote sensing, present unique challenges due to the complexity of the data and scarcity of annotations. These problems often require data from multiple sources, such as hyperspectral sensors on airplanes and multispectral sensors on satellites, which are expensive and time-consuming to acquire.This dissertation addresses the key question: how can large vision models be built and trained effectively under data constraints? The proposed solution involves integrating domain-specific knowledge into large vision models, specifically vision transformers, to optimize their performance and training efficiency. By incorporating core signal processing techniques, domain-specific knowledge is encoded as prior information, guiding the feature extraction process and refining randomly initialized queries via a query refiner module. This approach accelerates convergence with limited training data. Three key applications are explored: (1) methane detection in remote sensing from aerial imagery, (2) animal detection and classification in large grasslands for ecological studies, and (3) estimation of physiological signals such as ECG and ISTI for stress assessment in biomedical contexts. This research establishes an optimal methodology for embedding domain-specific knowledge into deep learning models, thereby enhancing performance in data-limited environments. It provides valuable insights for improving the applicability of vision transformer-based models across various domains, contributing to advancements in computer vision research and its practical real-world applications.

This research focuses on enhancing computer vision algorithms using eye tracking and visual saliency. Recent advances in eye tracking device technology have enabled large scale collection of eye tracking data, without affecting viewer experience. As eye tracking data is biased towards high level image and video semantics, it provides a valuable prior for object detection in images and object extraction in videos. We specifically explore the following problems in the thesis: 1) eye tracking and saliency enhanced object detection, 2) eye tracking assisted object extraction in videos, and 3) role of object co-occurrence and camera focus in visual attention modeling.

Since human attention is biased towards faces and text, in the first work we propose an approach to isolate face and text regions in images by analyzing eye tracking data from multiple subjects. Eye tracking data is clustered and region labels are predicted using a Markov random field model. In the second work, we study object extraction in videos using eye tracking prior. We propose an algorithm to extract dominant visual tracks in eye tracking data from multiple subjects by solving a linear assignment problem. Visual tracks localize object search and we propose a novel mixed graph association framework, inferred by binary integer linear programming. In the final work, we address the problem of predicting where people look in images. We specifically explore the importance of scene context in the form of object co-occurrence and camera focus. The proposed model extracts low-, mid- and high-level and scene context features and uses a regression framework to predict visual attention map. In all the above cases, extensive experimental results show that the proposed methods outperform current state-of-the-art.

The development of automated methods capable of detecting and localizing actions is crucial for a variety of applications, ranging from surveillance and autonomous driving to content moderation. This thesis focuses on creating action detection methods that deliver robust performances. At the heart of these methods’ robustness lie two fundamental elements: the detection of atomic actions and the ability for compositional understanding.

Atomic actions are those that are identifiable from a single image or a short video. In this research, we developed innovative methods to detect and localize such actions that achieve state-of-the art performance. The key strength of these methods lies in their ability to refine visual features both spatially and semantically, enabling precise identification of action-specific regions. For scalability, we further developed a multi-branch network to recognize new composition of objects and actions. Our design ensures that each branch learns decoupled features, allowing the network to transfer previously learned concepts to identify new compositions. This approach outperforms existing methods by a good margin as our extensive experiments on benchmark datasets demonstrate. Further, the correct identification of the attributes of the participating objects in actions helps to detect unknown compositions. Therefore, we have created a network utilizing spatially localized learning to correctly associate objects and attributes. This network achieves state-of-the-art performance in object-attribute association on cluttered scenes.

The developed methods in this thesis can do robust action detection at scale and serve as a base for numerous future applications.

Computer vision has made a significant progress in recent years thanks to advancement in neural network architectures and computing power. At the sensory level, the current machine vision systems sample the visual data uniformly to make predictions about the scene. This is in contrast with the human vision system that has high visual acuity only in a small central region, the fovea, and much coarser sampling away from the center. There has been a renewed interest, particularly in the context of active vision for robotics navigation and scene exploration, to develop biologically motivated methods that can leverage such foveated computations. While foveated vision offers computational savings at or near the region of interest, it requires eye movements to scan the scene for effective image understanding. The hypothesis is that methods that can leverage non-uniform sampling of the field of view together with eye-movements will lead to a new class of active vision systems that are optimized computationally for specific tasks of interest.

Inspired by the above observations, this research provides, for the first time, a comprehensive study of the human visual search in the constrained setting of person identification in the wild. A novel video database is created that systematically tests how different parts of a person contribute towards eye-movements and person identification. Our study shows that the search errors can dominate the overall recognition accuracy in human subject experiments. This calls for new strategies for integrating eye tracking with foveated image representations. Towards this two specific approaches are investigated further.

In the first approach, a deep neural network based method is developed to model eye movements. Using the long-short-term-memory to model the successive fixations. The proposed method outperforms state of the state of the art performance while simplifying the feature extraction procedure. The second approach focuses on the foveated image model that leverages multiple fixations. A convolutional neural network method is proposed that works directly with the foveated input images that achieves competitive recognition rates compared to standard neural networks operating on the same number of input pixels.

Overall the thesis investigates the requirements and implementations that could support active foveated vision, and lays down the ground work for future studies in this area.

Today's computer vision methods attempt to solve problems ranging from image classification to semantic segmentation. While some of these models are quite effective at their tasks, the most effective ones require a training set complete with a large set of heavily annotated data, but these large datasets and their annotations do not come without a cost. Dataset curators spend countless hours collecting data and even more data annotating it with semantic labels suitable for training today's methods. The expenses associated with data collection and annotation grow exponentially as the data becomes increasingly more scientific and difficult to annotate. While any lay person can take a photo of a dog and label it, collecting videos of the ocean floor and labelling the species in those videos can only be done with a budget sufficient to compensate a team of expert marine scientists. These costs motivate computer vision methods that can learn from less data, cheaper annotations, and less supervision. This thesis aims to provide some of these methods. We first introduce Point-supervised Class Activation Maps (PCAMs) to aid in semantic segmentation of images given only point level labels. Then, we introduce the Dataset for Underwater Invertebrate and Substrate Analysis (DUSIA), which comes with a limited set of partial labels. To address the challenges associated with learning from those labels, we train the Context Driven Detector with a Negative Region Dropping method, which enables better performance given partial labels. Finally, we introduce Context-Matched Collages as a means for generating additional training samples at a relatively low cost, leading to state of the art object detection performance on DUSIA.

Electron backscatter diffraction (EBSD) is a scanning electron microscopy technique used for collecting orientation properties of a material sample over space at the micro-meter scale. Because collecting this data is known for being costly and time-consuming, various methods have been proposed to upsample collected data, or generate new microstructures from a latent space. We propose a novel interpolation algorithm for quaternions that is imprevious to symmetry switching, named Minimum Angle Transformation Spherical Linear Interpolation (Slerp-MAT). We also propose a new Physics-based loss function based on on this algorithm, to obtain state-of-the-art results, in terms of the angular difference of the superresolved data and the ground truth. The result is a $882\%$ reduction in mean angular distance of Superresolved versus Ground Truth data for the collected Nickel dataset with respect to the previous state-of-the-art loss function, and a $321\%$ reduction for the collected Titanium dataset.