UC San Diego

Modern neural architectures present strong performance on various tasks, but their abilities are compromised when faced with the tasks requiring robust reasoning or involving abstract concepts. This dissertation starts with a study of the empirical representation power of two popular sequential neural networks, Long Short-Term Memory (LSTM) and Transformer network, in learning context-free grammar. The experiment results exhibit that the performance of both models is affected by the types of supervision and model regularization setting. Specifically, the LSTM and the Transformer network struggle to capture the grammatical structures, particularly the grammars with deeper recursions, when solely trained on input sequences in the generative language model setting. The performance gap between settings suggests two improvements when the input data structure is available. One approach is to supervise the model with intermediate states, while another is to regularize the models’ hidden states to enforce the its decomposition to store the state and to simulate the stack, mimicking a pushdown automaton, on separate segment.Complementing the study on sequential neural networks with prototypical grammars, several real-world applications are presented to showcase how the controlling the model complexity and intermediate state can enhance the performance. For the item retrieval problem in recommendation system, another transformer-based sequential model is proposed to capture the multiple interests of a user. Unlike the existing multi-interest frameworks, the proposed model is unbiased towards the site-wide popular items at the cost of incorporating a non-differentiable clustering method inside the neural network. The non-differentiable challenge are solved by two-stage training method, borrowing the idea from K-means clustering algorithm. In time series prediction tasks, a continuous convolutional neural network (CCNN) is proposed to handle non-uniformly sampled time series without preset interpolation kernels. CCNN can directly apply to the auto-regression tasks, and also works with temporal point process front-end to predict the time interval to the future event in the event-based time series. Lastly, a progressive optimization system, HISS , is designed to simplify the expressions in the intermediate representation of Halide compiler. HISS is built end-to-end without any human knowledge of the mathematical equivalences. Multiple training techniques, existing symbolic transformations of the expressions, and the evolving neural network modules, contribute to the HISS ’s performance that exceeds conventional searching method and human-heuristic methods. These three applications illustrate the benefits and techniques of applying the aforementioned philosophy, which suggests designing and supervising the neural networks to reflect the desired properties of neural networks. In conclusion, by presenting an empirical study method for evaluating the representation power of neural networks, which can be extended to broader areas, and the demonstrating how to design and overcome the challenges associated with symbolic properties in various real-world applications, this dissertation contributes to the advancement of the neural symbolic systems.