UC Berkeley

Modern vision models face a fundamental scaling challenge: real-world images and videos contain far more visual information than current models can process since they treat every pixel equally and encode all of them uniformly. High-resolution images contain small text, distant objects, and fine local details, while long videos multiply this burden across time. This thesis studies efficient methods and designs to scale vision models to extremely high spatiotemporal fidelity, inspired by the selective nature of human vision where high-fidelity processing is allocated unevenly across space, scale, and time.The first part of the thesis studies whether scaling vision models to higher fidelity is necessary. I show that Scaling on scales (S2 ), a naive approach of scaling up visual resolution by running frozen vision models on tiles of larger images, can already match or exceed the benefit of scaling model size across classification, dense prediction, multimodal language-model benchmarks, and robotic manipulation. The second part explores how to scale vision pre-training to extremely high resolution. I introduce PS3, a scale-selective vision-language pre-training method that enables pre-training on 4K-resolution images with near-constant cost by contrasting between local high-resolution regions and captions. The resulting PS3 and VILA-HD model improves high-resolution multimodal language-model performance, especially on the self-collected 4KPro benchmark that evaluates visual question answering that truly requires 4K perception. The third part discusses how to scale not only spatial, but also temporal fidelity for video models. I introduce AutoGaze, an autoregressive gazing module that removes redundant video patches before passing to expensive vision encoders or vision-language models, enabling efficient understanding of long, high-resolution videos, improving the performance on various video understanding benchmarks, especially on HLVid, the self-collected high-resolution long-form video understanding benchmark. Together, these contributions demonstrate the way of efficient visual scaling to extremely high spatiotemporal fidelity and achieving human-level visual understanding of fine-grained details across spatial and temporal dimensions.