UC Riverside

Mobile Augmented Reality (AR) is becoming more and more popular, with the AR market estimated to grow to $61 billion by 2023. However, there is a lack of understanding of AR performance in terms of accuracy, latency, among others. For example, a virtual object augmented on the table may drift in the air if the AR accuracy is low. The initialization latency for the multi-user AR can be long and will significantly impact user experience.This dissertation explores and improves the performance for both deep learning based and SLAM based AR on mobile devices on the network edge. First, we propose DeepDecision, a deep learning framework that ties together front-end devices with more powerful backend “helpers” (e.g., home servers) to allow deep learning to be executed locally or remotely in the cloud/edge server. The complex interaction between model accuracy, video quality, battery constraints, network data usage, and network conditions is considered to determine an optimal offloading strategy. Our results show DeepDecision achieves better accuracy comparing with the baseline method. Second, we developed a lightweight change detector which triggers deep neural networks(DNN) execution when there are significant changes in the input video. When there are no significant changes, DNN will not be triggered and a lightweight tracking algorithm will be applied to maintain previous DNN results. The change detector has high accuracy and very low latency. It helps DNN system to save energy and reach real-time processing without offloading. Third, we propose SPAR, a SPatially consistent AR framework for SLAM-based multi-user augmented reality. SPAR communicates efficiently by only sending the most relevant environment data while maintaining or even improving the accuracy. We also propose a geo-distance filter so that after the virtual object is initially resolved, SPAR can continue to optimize its accuracy. SPAR also has a preliminary automated tool to measure accuracy in both single-user and multi-user cases. Our results show that SPAR has better accuracy and initialization latency comparing with the baseline method.