UC Santa Barbara

Text has become a central interface in modern vision-language models. In visual generative models, text provides an intuitive mechanism for controlling image and video synthesis. In multimodal large language models, however, text can also become a shortcut that causes models to underutilize visual information. This dissertation studies these two roles of text: as a controller for visual generation, and as a crutch that reveals limitations in multimodal understanding. The first part of this dissertation investigates text-based controllability in visual generation. I study how natural-language instructions can guide image editing, image generation, and video synthesis across different forms of visual control. For image editing, I develop a multimodal large language model agent that translates user requests into executable editing operations in professional software. Since ground-truth edits are difficult to define, I further introduce a generalist reward model that evaluates editing results through multimodal reasoning and provides scalar feedback for reinforcement learning. For video generation, I study text-guided camera control, where natural-language camera descriptions specify complex camera motions. By synthesizing controllable training data and disentangling visual appearance from camera motion, this work enables more flexible video generation guided by both content and camera instructions. I also examine diffusion-based image generation and editing, showing how manipulating the diffusion process and attention mechanisms can improve content preservation, visual attribute control, and compositional faithfulness. The second part of this dissertation studies the limitations of multimodal understanding. Although multimodal large language models have achieved strong performance on many vision-language tasks, they often rely heavily on textual cues and struggle when essential information must be inferred from images. I analyze this imbalance through spatial reasoning and planning tasks, showing that current models can fail even in visually simple scenarios when textual descriptions do not provide sufficient guidance. I further explore training strategies that reduce excessive textual hints and encourage models to reason more directly from visual inputs. Together, these studies show that text can serve as a powerful control interface in vision-language models, while also revealing the risks of excessive reliance on text in multimodal understanding. This dissertation highlights the importance of building systems that use text effectively while developing stronger visual reasoning capabilities.