UCLA

Multimodality has elevated the utility of modern machine learning models. Through data-driven deep learning, multimodal neural networks learn generalizable and robust features, providing increased performance and capabilities while promising greater resilience to changing environments. Despite such capabilities, the increased complexity of these networks renders them susceptible to other variations that can manifest at inference time - ranging from dynamics during a system's deployment to its active runtime. These weaknesses of multimodal networks have not been adequately addressed in existing work, resulting in severe inefficiencies or subpar performance when subject to inference-time variations in real-world environments. This dissertation examines the weaknesses of multimodal networks across a variety of inference-time variations and diverse datasets, demonstrating that the efficient adaptation of internal weights constitutes a powerful and highly generalizable solution. First, we examine the challenge of deployment-time sensor perspective shift in distributed multimodal localization systems. Existing multimodal deep neural networks do not tolerate novel sensor perspectives arising during deployment, suffering from large accuracy degradations. We introduce FlexLoc, which combats sensor perspective shift by conditioning a portion of network weights on the sensors' pose information. FlexLoc improves zero-shot localization under unseen configurations of sensor pose by almost 50% in comparison to baselines. Second, we propose ADMN, which further explores inference-time adaptation by addressing variations that manifest during runtime in real-world multimodal systems, which may necessitate per-sample adaptation. Across a wide range of downstream tasks (e.g., audio-visual classification, gesture detection), we showcase how intelligent allocation of resources across modalities is critical for addressing runtime variations, ensuring optimal usage of limited computational resources. Third, we introduce SWAN, which extends the innovations of ADMN into the complex realm of multi-object detection in autonomous driving. We jointly explore adaptation to three forms of runtime variations -- changes in modality quality, platform resources, and complexity of the input sample itself. Moreover, we demonstrate how such a system can be practically realized on a real-world edge device. Finally, moving away from dedicated, task-specific models, we introduce CRAFT, an efficient method of adapting a general purpose multimodal foundation model at deployment time to an arbitrary downstream task. During this, we highlight the unique challenges associated with this class of neural networks. Ultimately, by leveraging the shared paradigm of context-aware model adaptation, these works collectively demonstrate that adjusting neural network parameters according to the state of the world is a highly effective strategy for mitigating inference-time variations.