UC San Diego

Event cameras, or dynamic vision sensors, offer distinct advantages over frame-based cameras, such as a high dynamic range, exceptional temporal resolution, and heightened sensitivity to motion. These characteristics make them particularly suitable for tasks like human pose estimation and dense trajectory estimation.

However, effectively utilizing the spatiotemporal features in event stream data remains challenging. This dissertation addresses this by tackling high-frequency 3D human pose estimation under dynamic lighting conditions, enabling the accumulation and flow of temporal information across time bins. Additionally, for dense continuous-time trajectory estimation, which requires precise pixel-level instantaneous speed understanding, a novel layered bidirectional time surface event representation is introduced. This improved event representation preserves abundant local speed cues, significantly enhancing the feature tracking process.

Throughout the exploration of these event camera-based vision tasks, a critical challenge emerges: the lack of sufficient data. As a novel vision modality, there are few large datasets with high-quality labels and paired data from other modalities. To advance this field, it is crucial to address this issue. This dissertation presents three solutions: generating event data using 3D simulation software, converting video data to event streams, and directly generating event streams using a generative model. These methods explore promising directions for event data generation and complement each other. In the final method, the event modality is aligned with mainstream modalities, allowing events to be generated from other modalities' inputs and vice versa. These efforts help bridge the domain gap between events and other modalities.