UCLA

Over the past decade, deep learning models have gained significant popularity for processing noisy sensor inputs in various real-world scenarios. However, they still face challenges such as data inefficiency, interpretability limitations, and weak reasoning abilities. On the other hand, Symbolic algorithms offer high-level reasoning and interpretability but struggle with uncertainties and high-dimensional sensor data. Neuro-symbolic approaches have emerged as promising solution, combining the strengths of deep learning and symbolic methods. They show potential for Complex Event Processing (CEP) in critical domains like healthcare monitoring and smart city control. This work explores the performance of neuro-symbolic methods in complex event detection tasks from multimodal sensor data, which involves recognizing complex events with complicated temporal patterns that span a wide time range. However, the lack of multimodal and complex event datasets poses a challenge. To address this, we formulate the complex event detection problem, synthesize a multimodal activity dataset, and build a stochastic daily human activity simulator to generate multimodal complex event datasets. We design a two-module system that includes a multimodal fusion module to handle sample inconsistencies and utilize information across modalities, and a backbone module that can be replaced with neural-only and neuro-symbolic models for performance comparison. The evaluation results demonstrate the superiority of the neuro-symbolic approach in complex event detection and provide insights for developing improved neuro-symbolic architectures for effective and applicable complex event detection systems in multimodal settings.