Non-linguistic Vocalization Recognition refers to the detection and classification of non-speech voice such as laughter, sneeze, cough, cry, screaming, etc. It could be seen as a subtask of Acoustic Event Detection (AED). Great progress has been made by previous research to increase the accuracy of AED. On the front end, multiple kinds of features such as Mel-Frequency Cepstral Coefficients (MFCCs), Gammatone Cepstral Coefficients (GTCCs) and many other hand-crafted features were explored. While on the back end, models or methods such as Gaussian Mixture Models (GMMs), Hidden Markov Models (HMMs), Bags-of-Audio-Words (BoAW), Support Vector Machine (SVM) and various types of neural networks were experimented.

Recent researches on Automatic Speech Recognition (ASR) and Acoustic Scene Classification (ASC) show the advantage of using Convolutional, Long Short-Term Memory, Deep Neural Networks (CLDNNs) on audio processing tasks. In this thesis, I am building a non-linguistic vocalization recognition system using CLDNNs. Log Mel-filterbank coefficients are adopted as input features and data augmentation methods such as random shifting and noise mixture are discussed. The built system is evaluated on a custom dataset collected from several resources and tested for real time application. The performance of CLDNNs for non-linguistic vocalization recognition is also compared with hybrid GMM-SVMs, Convolutional Neural Networks, Long Short-Term Memory and a fully connected Deep Neural Network trained on VGGish embeddings.

The results indicate that CLDNNs outperform the other models in classification precision and recall. Visualization of CLDNNs are presented to help understand the framework. The model is proved accurate and fast enough for real time applications.

Teaching machines to speak and act like a human is one of the longest-running goals in Artificial Intelligence. This thesis tackles two important problems in building the next generation of dialogue systems: enhancing the emotional intelligence of social chatbots and learning semantic structures from dialogue corpus. On the one hand, taking into account emotional quotient in dialogue system design help machine to mimic human behavior and further improve the long-term user engagement. On the other hand, extracting structural information from dialogue data is critical for us to analyze user behavior and system performance. The technology could be applied to various areas in computational linguistics, such as dialogue management, discourse analysis, and dialogue summarization.

This thesis consists of two parts. In the first part, we aim to present our efforts at studying emotional intelligence in dialogues systems. We break down the problem into three subjects: 1) the modeling and incorporation of human values, i.c., people tend to have common attitudes towards some statements or scenarios; 2) the inference of social relations between interlocutors from dialogues. Chatbots with such inferring capability can understand human behavior better and act appropriately; and 3) the modeling and tracking of speakers' mental states. This is beyond understanding what users say to perceive what users imply, requiring agents to mentally simulate the evolution status of the environment.

In the second part of this thesis, we investigate how we can extract structural information from dialogue corpus. In particular, we pioneered two research directions: 1) we reconstruct the original dialogues with variational recurrent neural networks and structured attention, then we extract the structure by computing the transitions between latent states; and 2) we detect and track the status of potential slot token groups to approximate a representation of task-oriented dialogue structures. We explored the problem from both theoretical and practical perspectives.