UC Merced

Neural networks generally require large amounts of data to adequately model thedomain space. In situations where the data are limited, the predictions from these models, which are typically obtained from stochastic gradient descent (SGD) minimization algorithms, can be poor. In addition, the data is commonly corrupted due to poor imaging appatus. In these cases, the use of more sophisticated optimization approaches and model architectures becomes crucial to increase the impact of each training iteration. Second-order methods can capture curvature information, providing a more informed guess on the direction and step length. However, they require vast amounts of storage and can be computationally time demanding. To address the computational issue, we propose an optimization algorithm that uses second-derivative information, exploiting curvature information for avoiding saddle points. We utilize a Hessian-free approach where we do not explicitly store the second-derivative matrix, by applying a conjugate gradient method. The algorithm uses a trust-region method, which does not require the Hessian to be positive definite. We present numerical experiments which demonstrate the improvement in classification accuracy using our proposed approach over a standard SGD approach. We propose using a limited-memory symmetric rank-one quasi-Newton approach which further addresses the time and space computational complexity. The approach allows for indefinite Hessian approximations, enabling directions of negative curvature to be exploited. Furthermore, we use a modified adaptive regularized using cubics approach, which generates a sequence of cubic subproblems that have closed-form solutions with suitable regularization choices and investigate the performance of our proposed and compare our approach to state-of-the-art first-order and other quasi-Newton methods. To incorporate the benefits of an exponential moving average algorithm to a quasi-Newton approach, we propose a quasi-Adam approach. Judicious choices of quasi-Newton matrices can lead to guaranteed descent in the objective function and improved convergence. In this work, we integrate search directions obtained from using these quasi-Newton Hessian approximations with the Adam optimization algorithm. We provide convergence guarantees and demonstrate improved performance through an extensive experimentation on a variety of applications. Finally, to mitigate the issue of data corruption, we propose a variety of architectures for various applications in image processing. We propose a blind source signal separator, which involves separating image signals which have been superimposed by a common observing apparatus. We propose novel deep learning architectures for low photon count image denoising, which contains Gaussian noise in a low-photon count setting. Then we propose a novel architecture for lowphoton count and downsampled imaging, where the signal is interfered with some Gaussian noise, Poisson noise and then downsampled. Finally, we propose a novel adversarial detection method for white-box attacks using Radial basis function and Discrete Cosine Transforms.