UC Santa Barbara

Photorealistic image synthesis has long been a central goal of computer graphics. Classical rendering pursues this goal by simulating light transport on an explicit scene description: it delivers precise artistic control and consistency, but falls short of ultimate photorealism and depends on costly assets and expensive light transport. Generative image and video models pursue the same goal differently. They learn the distribution of real photographs and videos and synthesize new samples directly, achieving high photo-realism, yet they lack precise control and consistency.In this dissertation, we combine the strengths of both approaches through generative rendering, using G-buffers – per-pixel scene quantities such as albedo, normals, roughness, metallicity, depth, and lighting – as a structured control interface between a rendering pipeline and a learned generative model. G-buffers are structured enough to carry precise artistic control, yet compact enough to leave a generative model room to synthesize the photorealism that is expensive to author and simulate. We call the resulting framework RGB↔X, where RGB denotes the final pixel image and X denotes the stack of G-buffer channels. The framework consists of two paired models: a forward model X→RGB that performs generative rendering by synthesizing an RGB image from G-buffers, and an inverse model RGB→X that performs inverse rendering by estimating G-buffers from an RGB image.In the first part, we develop an image-domain instance of RGB↔X that validates G-buffers as a practical intermediate medium through which users can controllably generate the images they want. The X→RGB model renders photorealistic images from G-buffer inputs, giving control over base color (albedo), geometry (normal), materials (rough-ness and metallicity), and lighting, for which we use diffuse irradiance as the control representation. Input G-buffers can be partial: users fix only the properties they care about and let X→RGB creatively fill in the rest. The paired RGB→X model estimates the same G-buffers from real RGB images, closing the loop between forward and inverse generative rendering and enabling editing applications such as material replacement and object insertion. Because no dataset pairs real photographs with G-buffers, both X→RGB and RGB→X in this first part are trained primarily on synthetic renderings. X→RGB therefore inherits a synthetic look that bounds its photorealism.In the second part, we push generative rendering toward high photorealism by training X→RGB directly on real data. The obstacle is the paired data bottleneck identified above. We resolve this by annotating large collections of real videos with our own RGB→X model to produce paired G-buffers and RGB images. We also label each training clip as real or synthetic, so that at inference time classifier-free guidance using these labels can push X→RGB generation toward photorealism. We introduce a set of new finetuning and streaming techniques on a pretrained diffusion transformer video backbone that accelerate convergence while avoiding catastrophic forgetting, yielding a unified RGB↔X framework for both forward and inverse rendering of images, videos, and streams, with flexible lighting control and long-sequence streaming support.Together, these results point toward image synthesis systems that retain the structure and controllability of classical rendering while achieving the photorealism learned by generative models from real data.