Search

Thesis
Peer Reviewed

High-Fidelity Spatial Signal Processing in Low-Power Mixed-Signal VLSI Arrays

Joshi, Siddharth
Advisor(s): Cauwenberghs, Gert

UC San Diego Electronic Theses and Dissertations (2017)

Machine learning and related statistical signal processing are expected to endow sensor networks with adaptive machine intelligence and greatly facilitate the Internet of Things (IoT). As such, architectures embedding adaptive and learning algorithms on-chip are oft-ignored by system architects and design engineers, and present a new set of design trade-offs. We focus on topologies efficiently implementing mixed-signal matrix-vector multiplication for applications in spatial filtering for IoT, where substantial processing gain in the analog domain alleviates the need for highly accurate and energy-consuming analog-to-digital conversion. We present a micropower, high-dynamic-range multichannel multiple-input multiple-out (MIMO) mixed-signal linear transform system, with analog signal path and digital coefficient control, composed of an array of 14-bit Nested Thermometer Multiplying DACs (NTMDACs) implementing analog multiplication, and variable gain amplifier (VGA) implementing accumulation. Implemented in 65nm CMOS, the NTMDAC MISO system-on-chip measures 84 dB in interference suppression at 2 pJ of energy per mixed-signal multiply-accumulate. We demonstrate state-of-the art performance on two tasks, spectrally oblivious interference suppression in communication signals and EEG signal separation. We then provide experimental demonstration of the use of a MIMO mixed-signal linear-transform system within a radio-frequency receiver chain. Over-the-air experiments demonstrating signal separation for two broad-band modulated signals further validate the adaptive beamforming capabilities under severe multipath conditions even in the absence of line-of-sight communication path.

In order to mitigate adverse effects of radix errors and capacitive mismatch encountered in compact low-power realizations of high-resolution, high-dimensional MIMO analog processing systems, we introduce Stochastic Successive Approximation, or S2A, as an on-line adaptive optimization algorithm amenable to efficient implementation in massively parallel analog hardware. S2A offers a direct alternative to stochastic gradient descent overcoming several of its shortcomings, such as its sensitivity to analog mismatch model errors, while improving on the rate of convergence for high-dimensional analog computation. The S2A algorithm enables convergence to values closer to the optimal when facing non-convex optimization landscapes induced by mismatch in capacitive multiplying digital-to-analog converter components when applied to adaptive analog signal processing. We experimentally demonstrate, in fewer than 25 iteratations of S2A, 65 dB of processing gain in adaptive beamforming, over-the-air, multipath interferer suppression.

Cover page: High-Fidelity Spatial Signal Processing in Low-Power Mixed-Signal VLSI Arrays

Article
Peer Reviewed

Memory Organization for Energy-Efficient Learning and Inference in Digital Neuromorphic Accelerators

UC Irvine Previously Published Works (2020)

The energy efficiency of neuromorphic hardware is greatly affected by the energy of storing, accessing, and updating synaptic parameters. Various methods of memory organisation targeting energy-efficient digital accelerators have been investigated in the past, however, they do not completely encapsulate the energy costs at a system level. To address this shortcoming and to account for various overheads, we synthesize the controller and memory for different encoding schemes and extract the energy costs from these synthesized blocks. Additionally, we introduce functional encoding for structured connectivity such as the connectivity in convolutional layers. Functional encoding offers a 58% reduction in the energy to implement a backward pass and weight update in such layers compared to existing index-based solutions. We show that for a 2 layer spiking neural network trained to retain a spatio-temporal pattern, bitmap (PB-BMP) based organization can encode the sparser networks more efficiently. This form of encoding delivers a 1.37x improvement in energy efficiency coming at the cost of a 4% degradation in network retention accuracy as measured by the van Rossum distance.

Cover page: Memory Organization for Energy-Efficient Learning and Inference in Digital Neuromorphic Accelerators

Creative Commons 'BY' version 4.0 license

Article
Peer Reviewed

Sub-μVrms-Noise Sub-μW/Channel ADC-Direct Neural Recording With 200-mV/ms Transient Recovery Through Predictive Digital Autoranging

Multicampus Research Programs and Initiatives (MRPI) Funded Publications (2018)

Cover page: Sub-μVrms-Noise Sub-μW/Channel ADC-Direct Neural Recording With 200-mV/ms Transient Recovery Through Predictive Digital Autoranging

Article
Peer Reviewed

Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines.

UC Irvine Previously Published Works (2016)

Recent studies have shown that synaptic unreliability is a robust and sufficient mechanism for inducing the stochasticity observed in cortex. Here, we introduce Synaptic Sampling Machines (S2Ms), a class of neural network models that uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised learning. Similar to the original formulation of Boltzmann machines, these models can be viewed as a stochastic counterpart of Hopfield networks, but where stochasticity is induced by a random mask over the connections. Synaptic stochasticity plays the dual role of an efficient mechanism for sampling, and a regularizer during learning akin to DropConnect. A local synaptic plasticity rule implementing an event-driven form of contrastive divergence enables the learning of generative models in an on-line fashion. S2Ms perform equally well using discrete-timed artificial units (as in Hopfield networks) or continuous-timed leaky integrate and fire neurons. The learned representations are remarkably sparse and robust to reductions in bit precision and synapse pruning: removal of more than 75% of the weakest connections followed by cursory re-learning causes a negligible performance loss on benchmark classification tasks. The spiking neuron-based S2Ms outperform existing spike-based unsupervised learners, while potentially offering substantial advantages in terms of power and complexity, and are thus promising models for on-line learning in brain-inspired hardware.

Article
Peer Reviewed

Stochastic Synapses Enable Efficient Brain-Inspired Learning Machines

UC Irvine Previously Published Works (2016)

Recent studies have shown that synaptic unreliability is a robust and sufficient mechanism for inducing the stochasticity observed in cortex. Here, we introduce Synaptic Sampling Machines (S2Ms), a class of neural network models that uses synaptic stochasticity as a means to Monte Carlo sampling and unsupervised learning. Similar to the original formulation of Boltzmann machines, these models can be viewed as a stochastic counterpart of Hopfield networks, but where stochasticity is induced by a random mask over the connections. Synaptic stochasticity plays the dual role of an efficient mechanism for sampling, and a regularizer during learning akin to DropConnect. A local synaptic plasticity rule implementing an event-driven form of contrastive divergence enables the learning of generative models in an on-line fashion. S2Ms perform equally well using discrete-timed artificial units (as in Hopfield networks) or continuous-timed leaky integrate and fire neurons. The learned representations are remarkably sparse and robust to reductions in bit precision and synapse pruning: removal of more than 75% of the weakest connections followed by cursory re-learning causes a negligible performance loss on benchmark classification tasks. The spiking neuron-based S2Ms outperform existing spike-based unsupervised learners, while potentially offering substantial advantages in terms of power and complexity, and are thus promising models for on-line learning in brain-inspired hardware.

Article
Peer Reviewed

Forward Table-Based Presynaptic Event-Triggered Spike-Timing-Dependent Plasticity

UC Irvine Previously Published Works (2016)

Spike-timing-dependent plasticity (STDP) incurs both causal and acausal synaptic weight updates, for negative and positive time differences between pre-synaptic and post-synaptic spike events. For realizing such updates in neuromorphic hardware, current implementations either require forward and reverse lookup access to the synaptic connectivity table, or rely on memory-intensive architectures such as crossbar arrays. We present a novel method for realizing both causal and acausal weight updates using only forward lookup access of the synaptic connectivity table, permitting memory-efficient implementation. A simplified implementation in FPGA, using a single timer variable for each neuron, closely approximates exact STDP cumulative weight updates for neuron refractory periods greater than 10 ms, and reduces to exact STDP for refractory periods greater than the STDP time window. Compared to conventional crossbar implementation, the forward table-based implementation leads to substantial memory savings for sparsely connected networks supporting scalable neuromorphic systems with fully reconfigurable synaptic connectivity and plasticity.

Cover page: Forward Table-Based Presynaptic Event-Triggered Spike-Timing-Dependent Plasticity

Article
Peer Reviewed

Memory-Efficient Synaptic Connectivity for Spike-Timing- Dependent Plasticity

UC Irvine Previously Published Works (2019)

Spike-Timing-Dependent Plasticity (STDP) is a bio-inspired local incremental weight update rule commonly used for online learning in spike-based neuromorphic systems. In STDP, the intensity of long-term potentiation and depression in synaptic efficacy (weight) between neurons is expressed as a function of the relative timing between pre- and post-synaptic action potentials (spikes), while the polarity of change is dependent on the order (causality) of the spikes. Online STDP weight updates for causal and acausal relative spike times are activated at the onset of post- and pre-synaptic spike events, respectively, implying access to synaptic connectivity both in forward (pre-to-post) and reverse (post-to-pre) directions. Here we study the impact of different arrangements of synaptic connectivity tables on weight storage and STDP updates for large-scale neuromorphic systems. We analyze the memory efficiency for varying degrees of density in synaptic connectivity, ranging from crossbar arrays for full connectivity to pointer-based lookup for sparse connectivity. The study includes comparison of storage and access costs and efficiencies for each memory arrangement, along with a trade-off analysis of the benefits of each data structure depending on application requirements and budget. Finally, we present an alternative formulation of STDP via a delayed causal update mechanism that permits efficient weight access, requiring no more than forward connectivity lookup. We show functional equivalence of the delayed causal updates to the original STDP formulation, with substantial savings in storage and access costs and efficiencies for networks with sparse synaptic connectivity as typically encountered in large-scale models in computational neuroscience.

Article
Peer Reviewed

Memory-Efficient Synaptic Connectivity for Spike-Timing- Dependent Plasticity.

UC Irvine Previously Published Works (2019)

Spike-Timing-Dependent Plasticity (STDP) is a bio-inspired local incremental weight update rule commonly used for online learning in spike-based neuromorphic systems. In STDP, the intensity of long-term potentiation and depression in synaptic efficacy (weight) between neurons is expressed as a function of the relative timing between pre- and post-synaptic action potentials (spikes), while the polarity of change is dependent on the order (causality) of the spikes. Online STDP weight updates for causal and acausal relative spike times are activated at the onset of post- and pre-synaptic spike events, respectively, implying access to synaptic connectivity both in forward (pre-to-post) and reverse (post-to-pre) directions. Here we study the impact of different arrangements of synaptic connectivity tables on weight storage and STDP updates for large-scale neuromorphic systems. We analyze the memory efficiency for varying degrees of density in synaptic connectivity, ranging from crossbar arrays for full connectivity to pointer-based lookup for sparse connectivity. The study includes comparison of storage and access costs and efficiencies for each memory arrangement, along with a trade-off analysis of the benefits of each data structure depending on application requirements and budget. Finally, we present an alternative formulation of STDP via a delayed causal update mechanism that permits efficient weight access, requiring no more than forward connectivity lookup. We show functional equivalence of the delayed causal updates to the original STDP formulation, with substantial savings in storage and access costs and efficiencies for networks with sparse synaptic connectivity as typically encountered in large-scale models in computational neuroscience.

Article
Peer Reviewed

Training a Probabilistic Graphical Model With Resistive Switching Electronic Synapses

UC Irvine Previously Published Works (2016)

Current large-scale implementations of deep learning and data mining require thousands of processors, massive amounts of off-chip memory, and consume gigajoules of energy. New memory technologies, such as nanoscale two-terminal resistive switching memory devices, offer a compact, scalable, and low-power alternative that permits on-chip colocated processing and memory in fine-grain distributed parallel architecture. Here, we report the first use of resistive memory devices for implementing and training a restricted Boltzmann machine (RBM), a generative probabilistic graphical model as a key component for unsupervised learning in deep networks. We experimentally demonstrate a 45-synapse RBM realized with 90 resistive phase change memory (PCM) elements trained with a bioinspired variant of the contrastive divergence algorithm, implementing Hebbian and anti-Hebbian weight updates. The resistive PCM devices show a twofold to tenfold reduction in error rate in a missing pixel pattern completion task trained over 30 epochs, compared with untrained case. Measured programming energy consumption is 6.1 nJ per epoch with the PCM devices, a factor of 150 times lower than the conventional processor-memory systems. We analyze and discuss the dependence of learning performance on cycle-to-cycle variations and number of gradual levels in the PCM analog memory devices.

Cover page: Training a Probabilistic Graphical Model With Resistive Switching Electronic Synapses

Article
Peer Reviewed

A compute-in-memory chip based on resistive random-access memory

UC San Diego Previously Published Works (2022)

Realizing increasingly complex artificial intelligence (AI) functionalities directly on edge devices calls for unprecedented energy efficiency of edge hardware. Compute-in-memory (CIM) based on resistive random-access memory (RRAM)¹ promises to meet such demand by storing AI model weights in dense, analogue and non-volatile RRAM devices, and by performing AI computation directly within RRAM, thus eliminating power-hungry data movement between separate compute and memory^2-5. Although recent studies have demonstrated in-memory matrix-vector multiplication on fully integrated RRAM-CIM hardware^6-17, it remains a goal for a RRAM-CIM chip to simultaneously deliver high energy efficiency, versatility to support diverse models and software-comparable accuracy. Although efficiency, versatility and accuracy are all indispensable for broad adoption of the technology, the inter-related trade-offs among them cannot be addressed by isolated improvements on any single abstraction level of the design. Here, by co-optimizing across all hierarchies of the design from algorithms and architecture to circuits and devices, we present NeuRRAM-a RRAM-based CIM chip that simultaneously delivers versatility in reconfiguring CIM cores for diverse model architectures, energy efficiency that is two-times better than previous state-of-the-art RRAM-CIM chips across various computational bit-precisions, and inference accuracy comparable to software models quantized to four-bit weights across various AI tasks, including accuracy of 99.0 percent on MNIST¹⁸ and 85.7 percent on CIFAR-10¹⁹ image classification, 84.7-percent accuracy on Google speech command recognition²⁰, and a 70-percent reduction in image-reconstruction error on a Bayesian image-recovery task.

Cover page: A compute-in-memory chip based on resistive random-access memory