UC Irvine

Estimating the temporal state of a system from image sequences is an important task for many vision and robotics applications. A number of classical frameworks for state estimation have been proposed, but often these methods require human experts to specify the system dynamics and measurement model, requiring simplifying assumptions that hurt performance. With the increasing abundance of real-world training data, there is enormous potential to boost accuracy by using deep learning to learn state estimation algorithms, but there are also substantial technical challenges in properly accounting for uncertainty. We propose solutions to these challenges by developing end-to-end differentiable particle based solutions which can accurately model uncertainties and allow for embedding of neural networks in our algorithms.

Concretely, we first create an end-to-end learnable particle filter that uses flexible neural networks to propagate multimodal, particle-based representations of state uncertainty. Our gradient estimators are unbiased and have substantially lower variance than existing, differentiable (but biased) particle filters. We apply our end-to-end learnable particle filter to the difficult task of visual localization in unknown environments, and show large improvements over prior work. We then expand on our particle filtering method to create the first end-to-end learnable particle smoother, which incorporates information from future as well as past observations, and apply this particle smoother to the real-world task of city-scale geo-localization using camera and planimetric map data. We compare to state-of-the-art baselines for visual geo-localization, and again show superior performance. Finally, we develop an end-to-end learnable particle belief propagation algorithm for inference on tree-structured graphical models, where the state of each node evolves over time based on unknown system dynamics. A key application of differentiable particle belief propagation is learning to estimate the articulated pose of dynamic human bodies.