UC Riverside

Deep networks have received considerable attention in recent years due to their applications in different problems of science and engineering. This dissertation explores the application of deep networks in continual learning and inverse problems. In this work, we enforce some simple structures on the networks to achieve better solution in terms of performance, memory and computational cost.

Continual Learning with Low-rank Increment: Continual learning is a process of training a single neural network on multiple tasks one after another, where training data for each task is often available only during the training of that task. Neural networks tend to forget older tasks when they are trained for the newer tasks; this property is often known as catastrophic forgetting. To address this issue, continual learning methods use episodic memory, parameter regularization, masking and pruning, or extensible network structures. This work proposes a continual learning framework based on low-rank factorization of the network weights. To update the network for a new task, a rank-1 (or low-rank) matrix is learned and added to the weights of every layer. An additional selector vector is also introduced that assigns different weights to the low-rank matrices learned for the previous tasks. Our proposed approach demonstrates superior performance compared to the current state-of-the-art methods with much lower number of network parameters.

Inverse Problems with Deep Networks: Inverse problems form a family of problems where we try to recover the true signal given the modified version of the signal. Since inverse problems are often ill-posed in nature, we often need to impose some constraints on the solution set. This dissertation mainly focuses on deep generative networks as a prior for solving inverse problems. Low-rank matrix and tensor structures have been used in this work as constraints on the input latent vectors of the deep generative networks to improve quality of the reconstruction with reduced parameters. This dissertation also explores unrolled networks where classical iterative solution approaches are structured as fixed layer networks with each iteration forming a layer of the network. We use such unrolled network structures to design sensing parameters for nonlinear inverse problems that led to achieving good reconstruction quality with a fixed number of layers (or iterations).