UC San Diego

The recent advances in Deep Learning (DL) have changed the landscape of Machine Learning significantly. However, over the last few years, we observe that to enable complex applications with DL, an increasing amount of compute resources are required for deployment. This has several undesirable effects ranging from increased deployment costs and energy consumption to harmful environmental effects. Recently, as the community continues to explore alternate paradigms for machine learning, Hyperdimensional Computing (HDC) has gained popularity, due to its low-footprint, low energy consumption, and ease of acceleration on parallel hardware. However, prior work has shown that HDC is not sufficiently accurate on a few complex applications. This dissertation explores the synergy between DL and HDC for improving the compute efficiency of learning while keeping state-of-the-art accuracy. We propose novel hybrid models that leverage the salient properties of HDC and DL to address each methods drawbacks.

We first introduce FHDnn, a hybrid architecture for federated learning that leverages HDC to improve compute efficiency and robustness to unreliable network conditions in IoT Networks. Our architecture uses a frozen pre-trained CNN for feature extraction with a HDC classifier that is trained in a federated manner. By training and transmitting only the HDC classifier, we avoid having to transmit large DNN models and consequently mitigate the robustness issues. Our experiments show that our proposed methods improve communication efficiency by 66$\times$ and are 6$\times$ faster than federated learning with DNNs. We then extend this work to analyze the convergence properties of FHDnn and show that FHDnn provides convergence guarantees for federated learning.

Next, we propose systematic approaches for designing hybrid architectures for different modalities of data. We leverage the binding operator in HDC to capture relationships between data and generate HDC representations that capture these relationships in a single vector. We then use these informative data representations as input to DNN models. This facilitates ease of learning as the model no longer needs to learn relationships in data from scratch. We first consider text data for multi-label classification and demonstrate on large-scale real-world datasets that our proposed architecture is 231$\times$ smaller and 16$\times$ faster compared to SoA models.

Finally, we propose a novel architecture for image classification that leverages the symbolic properties of HDC to structure the problem hierarchically to reduce learning complexity. We construct a label hierarchy by grouping together similar images into groups and generate label representations using HDC, that captures these group relationships. This allows us to break down the classification into 2 stages: detecting the group the image belongs to and identifying the specific label within the group. We show that this improves efficiency by up to 200$\times$ compared to SoA models with minimal loss in accuracy.