UC Riverside

Multimodal learning seeks to develop models capable of reasoning over heterogeneous inputs such as vision, language, audio, and sensor data. Despite significant progress in recent years, three key challenges remain: (i) supporting arbitrary combinations of input modalities, (ii) maintaining robustness when modalities are missing at inference time, and (iii) integrating capabilities from multiple expert models without costly joint retraining on large paired multimodal datasets. This dissertation addresses these challenges through a series of principled methods that progressively relax assumptions on modality availability, supervision, and data requirements. First, we study multimodal fusion under heterogeneous modality combinations. We propose MMSFormer, a Multimodal Segmentation Transformer with a novel fusion strategy that effectively integrates information from diverse modalities. Unlike prior approaches limited to fixed modality sets, MMSFormer supports arbitrary modality combinations and exhibits monotonic performance improvements as additional modalities are introduced. Extensive experiments demonstrate state-of-the-art performance on multimodal material and semantic segmentation tasks, with ablation studies highlighting the contribution of each modality. Second, we address robustness to missing modalities. We introduce two complementary approaches: (i) a parameter-efficient adaptation mechanism that modulates intermediate representations of pretrained models using fewer than 1% additional parameters, and (ii) Cross-Modal Proxy Tokens (CMPTs), which approximate missing modality representations by attending to available modalities without explicit modality generation. Both methods significantly reduce performance degradation across diverse tasks and settings while preserving strong performance when all modalities are present during inference. Finally, we propose Singular Subspace Alignment and Merging (SSAM), a training-free framework for combining independently trained multimodal large language models into a unified model. SSAM aligns modality-specific representations within a shared low-rank subspace and merges them in parameter space to preserve complementary knowledge while minimizing modality interference. Without using any paired multimodal training data, SSAM achieves state-of-the-art performance across multiple benchmarks, surpassing prior training-free merging methods and, in some cases, jointly trained multimodal models. Together, these contributions advance multimodal learning toward flexible, robust, and scalable systems capable of operating under real-world constraints.