UCLA

In recent years there has been an explosion in the availability and use of mobile, wearable, and Internet-of-Things (IoT) devices which generate vast volumes of sensory data. Moreover, deep learning techniques have shown excellent performance on a wide range of tasks such as visual understanding, natural language processing, and speech recognition. Inspired by the success of deep neural networks and the abundance of sensory data, researchers got increasing opportunities to adopt the successful deep learning techniques to various sensing applications. However, real-life sensing tasks usually involve complex spatiotemporal dependencies, which can hardly be captured using only a data-driven approach. The dynamic nature of the sensing environments exacerbates this problem, especially when labeled data are scarce. Unlike visual and natural language data, raw sensor data are opaque to humans, and cannot be labeled retrospectively in a crowdsourced manner. Therefore, training and deploying deep learning models for complex sensing tasks in dynamic scenarios remains a challenge.

In this dissertation, we first focus on the problem of complex event detection over heterogeneous sensory data. To address this problem, the designed system requires not only the perception ability for extracting informative and useful features from raw data, but also the reasoning ability for mining and analyzing the dependencies between the higher-level concepts. We propose DeepCEP, a neural-symbolic framework that combines the power of deep learning models and Complex Event Processing (CEP) engines. DeepCEP encodes prior knowledge provided by the users to help make effective inferences over the long-term, state-based complex events. To enable the training of this neural-symbolic system, we introduce Neuroplex, which learns from scratch efficiently in complex event settings, under the guidance of high-level prior knowledge. Compared with mainstream deep learning models, Neuroplex reduces the data annotation requirement by 100 times and speeds up the learning process by four times.

Furthermore, we propose the generalized framework DeepSQA for flexible inferencing over heterogeneous sensory data. Given a sensory data context and a task defined on the runtime as natural language questions about the data, DeepSQA can provide an accurate natural language answer. In addition to the DeepSQA, we create SQAGen, a software framework for generating SQA datasets using labeled source sensory data, and also generate OppQA with SQA-Gen for benchmarking different SQA models.

Lastly, we investigate the transferability and adaptability of deep learning models under dynamic environments. We propose RecycleML, a unified framework that transfers knowledge among deep learning models of IoT devices with different input modalities, deployments, or tasks. Using human activity recognition as a case study, over our collected CMActivity dataset, we observe that RecycleML reduces the amount of required labeled data by at least 90% and speeds up the training process by up to 50 times.