UC San Diego

Large-scale pretraining has produced powerful deep learning representations, but real-world data rarely match the clean, uniform assumptions of standard transfer learning. This dissertation uses heterogeneous data to describe settings in which information differs in modality, representation, relevance, quality, or frequency across inputs, outputs, tasks, or supervision. Examples include molecules represented as graphs, images, strings, and text; images containing both informative and redundant regions; noisy or incomplete labels; and rare but important classes. Applying pretrained models in these settings requires deciding which information to process, how to connect different forms of information, and how much to trust each supervision signal. This dissertation develops adaptive learning as a unifying approach through three paradigms: adaptive selection focuses computation on useful input components; adaptive alignment maps heterogeneous modalities, tasks, and outputs into a shared predictive space; and adaptive supervision adjusts imperfect or imbalanced training signals. The first contribution, EViT, demonstrates adaptive selection for efficient Vision Transformers. Standard Vision Transformers devote equal computation to all image patches despite substantial redundancy. EViT uses class-token attention to identify important tokens, preserves them in subsequent layers, and fuses inattentive tokens into a compact representation rather than discarding their context. This progressive token reorganization reduces computation while maintaining accurate visual recognition. The second contribution, DrugChat, demonstrates adaptive alignment for multimodal molecular prediction. Molecules can be represented as chemical graphs, rendered images, SMILES strings, and natural-language prompts, while predictions may require labels, probabilities, categorical decisions, or free-form descriptions. DrugChat encodes molecular graphs and images with pretrained encoders, projects them into the latent space of an instruction-following language model, and conditions generation on molecular structure and task prompts. It thereby aligns heterogeneous inputs in a shared representation space and expresses diverse prediction targets through a common language-generation interface. The third contribution, RobPicker, demonstrates adaptive supervision for cryo-electron tomography particle picking, where annotations are expensive, incomplete, noisy, and highly imbalanced across macromolecular species. Its meta-learning framework jointly trains a segmentation network, a label correction network, and a data reweighting network. Guided by cleaner validation data, these components refine imperfect labels and adapt each subtomogram's contribution to the loss, giving greater influence to rare or small particle classes. Together, EViT, DrugChat, and RobPicker apply adaptation at complementary levels of the learning pipeline: selecting what data to process, aligning heterogeneous representations and outputs, and learning from imperfect supervision.