UC Riverside

Optical flow problem focuses on predicting motion between consecutive frames, typically within the RGB domain. While RGB-based models have reached a high level of maturity, optical flow prediction in alternative modalities, such as thermal imagery, remains underexplored. Given that thermal imaging excels in challenging conditions like fog and darkness, developing a thermal optical flow model is of critical importance. We propose a first-of-its kind data-driven approach to optical flow, with a novel cross-modal supervision. To achieve this, we collected more than a thousand, spatially aligned and synchronized RGB and thermal video sequences across diverse environments and types of motion, including humans, vehicles, animals, and nature. We then leveraged this dataset to generate pseudo-ground-truth optical flow estimates by using state-of-the-art optical flow techniques on the RGB image, and then using them to fine-tuning the optical flow network with thermal images as input. Our contribution culminates in a high-quality, synchronized RGB-Thermal video dataset, with state-of-the-art baselines for optical flow in the thermal domain.