This thesis aims to achieve better automatic speech recognition (ASR) for children. The most challenging problem is a lack of transcribed available databases, and thus this problem could be regarded as a low-resource task. We introduce this problem from three aspects. First, compared to adults' speech, larger intra- and inter-speaker variabilities in children's speech exacerbate the low-resource problem due to the different growth patterns of children's vocal tracts. Second, there are children's speech data on the Internet untranscribed. Thus, exploring how to utilize such untranscribed data to improve children's ASR is significant but challenging. Last, lacking training data will cause inadequate training when using random model initialization. Therefore, finding a good model initialization is important for training a robust children's ASR model in a low-resource setting.

We improve the performance of children's ASR systems from the aforementioned three aspects. First, we compare multiple effective data augmentation methods for children's ASR. On the OGI Kids' Corpus, we can achieve a WER reduction of around 10 % for the HMM-BLSTM modeling-based hybrid ASR system and around 25 % for the Connectionist Temporal Classification - Attention-based Encoder-Decoder (CTC-AED) modeling-based end-to-end ASR system. Second, unsupervised pre-training and semi-supervised learning are used as two effective methods for utilizing untranscribed data. Using one of the unsupervised pre-training methods, bidirectional autoregressive predictive coding, and 3 iterations of semi-supervised learning could bring a WER reduction of 9.6 % with 60 hours of untranscribed data. Third, model-agnostic meta-learning (MAML) based meta-initialization (MI) was used to find a good model initialization. However, MI is vulnerable to overfitting on training tasks (learner overfitting). To alleviate learner overfitting, an age-based task-level augmentation method is proposed. After using task-level augmentation methods with MI, the children's ASR system is able to achieve a WER reduction of 51 % in kindergarten-aged speech over no augmentation or initialization.

A speaker's voice constantly varies in everyday situations, such as when talking to a friend, reading aloud, talking to pets, or narrating a happy incident. These changes in speaking style affect human and machine abilities to distinguish speakers based on their voice. This dissertation studies the effects of speaking style variability on speaker discrimination performance by humans and machines.

We compare human speaker discrimination performance for read speech versus casual conversations. Listeners perform better when stimuli are style-matched, particularly in read speech -- read speech trials. They perform the worst in style-mismatched conditions. Moderate style variability affects the "same speaker" task more than the "different speaker" task. The speakers who are "easy" or "hard" to "tell together" are not the same as those who are "easy" or "hard" to "tell apart." Analysis of acoustic variability suggests that listeners find it easier to "tell speakers together" when they rely on speaker-specific idiosyncrasies and that they "tell speakers apart" based on their relative positions within a shared acoustic space.

The effects of style variability on automatic speaker verification (ASV) systems are systematically analyzed using the UCLA Speaker Variability database, which comprises multiple speaking styles per speaker. The performance is better when enrollment and test utterances are of the same style, but it substantially degrades when styles are mismatched. We hypothesize that between-frame entropy can capture style-related spectral and temporal variations. We propose an entropy-based variable frame rate (VFR) technique to address style variability in two different approaches: data augmentation and self-attentive conditioning. Both approaches improve performance in style-mismatch scenarios and are comparable in performance.

Furthermore, humans and machines seem to employ different approaches to speaker discrimination. In an attempt to improve ASV performance in the presence of style variability, insights learnt from the human speaker perception experiments are used to design a training loss function, referred to as "CllrCE loss". CllrCE loss focuses on both speaker-specific idiosyncrasies and relative acoustic distances between the speakers to train the ASV system. This loss function improves ASV performance in case of style variability, especially in the case of moderate style variations from conversational speech.

The subglottal system comprises the trachea, bronchi and their accompanying airways. Its configuration changes very little compared to that of the supraglottal vocal tract, as a result of which its acoustic properties are relatively more stationary and speaker specific. In this dissertation, our knowledge of subglottal acoustics - subglottal resonances (SGRs), most importantly - is leveraged to develop novel solutions to three problems that involve using or estimating speaker-specific characteristics: (1) body height estimation, (2) speaker normalization for automatic speech recognition (ASR), and (3) speaker identification (SID) and verification (SV). The focus is on scenarios where purely statistical methods may be sub-optimal owing to limited and/or noisy speech data.

Simultaneous recordings of speech and subglottal acoustics are collected (using a microphone and an accelerometer, respectively) from native American English speakers (50 adults and 43 children) and 6 adult bilingual speakers of Mexican Spanish (first language) and American English. The data are analyzed to understand the relationships between SGRs, and vocal-tract resonances (formants), body height and native language. Results indicate that (1) phonological vowel features (tongue height and backness) can be characterized via acoustic measures of formants and SGRs, (2) SGRs correlate well with body height, and (3) SGRs are practically independent of language and phonetic content. Based on these findings, algorithms are developed for the automatic estimation of SGRs from speech signals (i.e., without using accelerometer information). The algorithms are found to be effective for both adults and children, in quiet as well as noisy environments; their performance is equally good for native English and bilingual English/Spanish speakers, and does not degrade much with limited data.

Predictive models between body height and SGRs (in conjunction with SGR estimation algorithms) are used to develop an automatic approach to speech-based height estimation for adult speakers. The method is comparable in performance to existing data-driven techniques, but requires less training data, offers better generalization, and is more robust to noise. In the context of ASR for children, SGRs are used for speaker normalization via piece-wise linear frequency warping. On a digit-recognition task, the method achieves lower word error rates than conventional vocal-tract length normalization, in clean as well as noisy environments. The benefit is particularly significant for young speakers (6-8 years old) and short utterances (1 or 2 words). For SID and SV (with adults' speech), an algorithm is developed for deriving subglottal features that are more informative (than SGRs) with regard to speaker discriminability. When combined with Mel-frequency cepstral coefficients (conventional speech features for SID and SV), subglottal features provide significant performance improvements, especially for short test utterances (5-10 seconds in duration).

The potential of speech technology to improve educational outcomes has been a topic of great interest in recent years. For example, automatic speech recognition (ASR) systems could be employed to provide kindergarten-aged children with real-time feedback on their literacy and pronunciation as they practice reading aloud. Within these systems, speaker identification (SID) technology could additionally be used to identify the user’s speaker characteristics in order to ensure that they receive age, language, and dialect-appropriate feedback. While these technologies are more established for well-represented groups in STEM (ie. able-bodied, adult, first-language speakers of mainstream dialects), they give much worse performance for underrepresented groups (young children, speakers of non-mainstream dialects, people with speech-related disabilities, etc.). This work focuses on improving speech technology performance for children’s speech and African American English (AAE) dialect speech with the goal of creating more equitable outcomes in early education. The contributions of this work span three primary areas: 1) Dialect identification and density scoring, 2) data augmentation for speech recognition, and 3) Natural Language Processing for fair and inclusive automatic speech assessment.

First, we create a robust system for dialect identification of African American English for both children and adult's speech. This system aims to take an input utterance from a speaker of either African American English or Mainstream American English and determine which of the two dialects the utterance belongs. The system fuses features from paralinguistics, self-supervised learning representations, automatic speech recognition system outputs, prosodic contours, and other descriptors of the speech signal in order to learn a mapping from the input acoustic information to a dialect classification decision. We further explore this architecture in automatic dialect density estimation, a task we create and develop. In dialect density scoring, we train a system to automatically predict a speaker's frequency of usage of dialect-specific patterns. This information can then be passed to a speech recognition system for more dialect-informed processing.

Second, we develop a data augmentation algorithm to improve zero-shot and few-shot speech recognition of low-resource dialects. The algorithm, named LPCAugment, deconstructs an input speech signal into a source and filter representation using linear predictive coding (LPC) analysis. The poles of the filter representation can then be perturbed independently of the source representation in order to model formant shifts that may be seen across accents and dialects. We use this perturbation method to artificially generate speech samples with shifted formant locations to serve as additional training data for a speech recognition system. This speech recognition system is then evaluated on children's speech for child speakers of a Southern California dialect and child speakers of an Atlanta, Georgia, area dialect.

Third, we explore automatic analysis and scoring of speech recognition transcripts for educational assessments. Given information about a student's spoken dialect and automatically generated transcripts of their oral response to an assessment prompt, we train a system to automatically grade the quality of the response with respect to a pre-determined criterion. This system uses language modeling and spoken information retrieval to identify key features in the spoken response and holistically decide if the response aligns with the grading criteria. Combined, the steps in this work form a framework for inclusive spoken language understanding technology that can be used to perform provide students with dialect-appropriate language training or language assessment.

This dissertation focuses on robust signal processing algorithms for birdsongs and speech signals. Automatic phrase or syllable detection systems of bird sounds are useful in several applications. However, bird-phrase detection is challenging due to segmentation error, duration variability, limited training data, and background noise. Two spectrograms with identical class labels may look different due to time misalignment and frequency variation. In real recording environments such as in a forest, the data can be corrupted by background interference, such as rain, wind, other animals or even other birds vocalizing. A noise-robust classifier needs to handle such conditions. Similarly, Automatic Speech Recognition (ASR) works well in quiet environments, but a large degradation in performance is observed when the speech signal is corrupted by background noise. The ASR performance would benefit from robust representations of speech signals and from robust recognition systems.

The first topic of this dissertation focuses on an automatic birdsong-phrase recognition system that is robust to limited training data, class variability, and noise. The algorithm comprises a noise-robust Dynamic-Time-Warping (DTW)- based segmentation and a discriminative classifier for outlier rejection. The algorithm utilizes DTW and prominent (high energy) time-frequency regions of training spectrograms to derive a reliable noise-robust template for each phrase class. The resulting template is then used for segmenting continuous recordings to obtain segment candidates whose spectrogram amplitudes in the prominent regions are used as features to a Support Vector Machine (SVM). In addition, we present a novel approach to training HMMs with extremely limited data. First, the algorithm learns the Global Gaussian Mixture Models (GMMs) for all training phrases available. GMM parameters are then used to initialize state parameters of each individual model. The number of states and the mixture components for each state are determined by the acoustic variation of each phrase type. The (high-energy) time-frequency prominent regions are used to compute the state emitting probability to increase noise-robustness.

The second topic of the dissertation deals with noise-robust processing for automatic speech recognition. We also propose a new pitch-based spectral enhancement algorithm based on voiced frames for speech analysis and noise-robust speech processing. The proposed algorithm determines a time-warping function (TWF) and the speaker's pitch with high precision, simultaneously. This technique reduces the smearing effect in between harmonics when the fundamental frequency is not constant within the analysis window. To do so, we propose a metric called the harmonic residual which measures the difference between the actual spectrum and the resynthesized spectrum derived from the linear model of speech production with various combinations of TWF and high-precision pitch values as parameters. The TWF and pitch pair that yields the minimum harmonic residual is selected and the enhanced spectrum is obtained accordingly. We show how this new representation can be also used for automatic speech recognition by proposing a robust spectral representation derived from harmonic amplitude interpolation.

The voice source contains important lexical and non-lexical information. The non-lexical information can convey, for example, prosodic events, emotional status, as well as cues pertaining to the uniqueness of the speaker's voice. A better understanding, and eventually a better model of the voice source, would benefit various speech applications, such as speech recognition, speech synthesis, speaker identification, age/gender classification, as well as clinical assessments.

This dissertation has three main goals. The first is to better understand the voice source through analyzing images of the vocal folds using laryngeal high-speed videoendoscopy (HSV) recordings. A new automatic method is proposed to compactly summarize the overall spatial synchronization pattern of vocal fold vibration for the entire laryngeal area from HSV data. Additionally, a new measure is proposed to adequately capture perceptually-important variations in glottal area pulse shapes, which are extracted from HSV data.

The second goal is to study the acoustic consequence of a physiological vocal-fold vibration pattern---the glottal gap effect, and apply our findings to a gender classification task of children's voices. Voice source related measures are found to improve classification accuracy, especially for younger (10-15 year old) speakers.

The third goal is to propose new voice source models and evaluate them in different applications. In the first application, a new source model and a noise-robust automatic source estimation algorithm are proposed to estimate the voice source from speech signals. Results in both clean and noisy conditions show that the proposed model and algorithm are robust in accurately estimating the voice source signal. The second application is to use the proposed source model for vowel synthesis. Perceptual listening experiments show that the proposed model provides a better perceptual match to the target voice than do traditional models.

Automatic depression detection systems based on speech signals have recently garnered significant attention. Depression modeling from speech signals, however, faces three challenges. The first challenge is data scarcity. The second, is the risk of privacy exposure in depression detection systems. The third, is the lack of consideration of non-uniformly distributed depression patterns within speech signals. In this dissertation, we address these challenges so that better and privacy-preserving speech-based depression detection systems are built.

To address the data scarcity issue, we propose a modified Instance Discriminative Learning (IDL) pre-training method to enable the model to extract augment-invariant and instance-spread-out embeddings from pre-training tasks using out-of-domain unlabeled data. The pre-trained model is then used for initialization of the downstream model, DepAudioNet, and fine-tuned for depression detection tasks. We investigate different augmentation techniques and instance sampling strategies in the pre-training stage. Specifically, we propose a novel sampling strategy, Pseudo-Instance-based Sampling (PIS), to further reveal the correlation between depression characteristics with the underlying acoustic units.Second, to address the privacy-preservation issue, we propose a novel non-uniform speaker disentanglement (NUSD) adversarial learning framework to disentangle speaker-identity information from depression characteristics. The approach utilizes idiosyncratic behaviors of different layers of detection models, and varies the adversarial disentanglement strength of different model components. The method shows that depression detection can be done without an over-reliance on speaker-identity features. More importantly, we found that attenuating more speaker information in the Feature Extraction (FE) module yields better performance than assigning the disentanglement weights uniformly.

Third, to address the non-uniformity of depression patterns in speech signals, we propose a novel framework. The framework, Speechformer-CTC, models dynamically varying depression characteristics within speech segments using a Connectionist Temporal Classification (CTC) objective function without the necessity of input-output alignment. Two novel CTC-label generation policies, namely the Expectation-One-Hot and the HuBERT policies, are proposed and incorporated in objectives at various granularities. Additionally, experiments using Automatic Speech Recognition (ASR) features are conducted to demonstrate the compatibility of the proposed method with content-based features. Our findings show that depression detection can benefit from modeling non-uniformly distributed depression patterns and the proposed framework can potentially be used to determine significant depressive regions in speech utterances.

Experiments show that the proposed techniques achieve state-of-the-art performance and are validated for both English and Mandarin Chinese.

Pitch (F0) detection has been a very active research topic in recent years. Accurate F0 detection is very important in applications such as speech recognition and coding. This work examines the effects of demodulating a signal into its modulator and carrier parts, each bearing low and high frequency content of the signal respectively, on pitch detection accuracy. Since the carrier part is known to contain the high frequency pitch content of a speech signal, using it as an input to pitch detection algorithms proves to be beneficial in most of the tests carried out with different speakers, and under different noise levels.

Automatic speech recognition (ASR) systems have improved significantly in the last decade due to advances in deep learning algorithms and easier access to very large databases. ASR systems, however, face two major challenges. The first challenge is in low-resource situations, such as child speech, where the accuracy degrades significantly, and the second challenge concerns low inference efficiency due to the autoregressive mechanism and size of ASR models. In this dissertation, we address these challenges by introducing novel techniques to improve the accuracy and the inference efficiency for ASR tasks, especially child ASR.

To address the accuracy challenge, we introduce novel self-supervised learning (SSL) methods using un-annotated adult speech data and explore how these methods can improve the downstream child ASR tasks. Specifically, a bidirectional autoregressive predictive coding (Bi-APC) method is proposed for non-causal models pretraining with the usage of adult speech data. The pretrained model is then finetuned on supervised child speech-text pairs. We also propose a novel framework, domain responsible adaptation and finetuning (DRAFT), to reduce the domain shifting in pretrained speech models. The DRAFT framework is effective for APC that uses a causal transformer as the backbone, and for Bi-APC, Wav2vec2.0 and HuBERT methods, which use a non-causal transformer as the backbone.

To address the inference efficiency challenge, we introduce a novel Connectionist Temporal Classification (CTC) Alignment-based Single-Step Non-Autoregressive Transformer (CASS-NAT) for end-to-end ASR. A comprehensive evaluation of CASS-NAT is performed in this dissertation. In CASS-NAT, the word embeddings in an autoregressive transformer (AT) are substituted with token-level acoustic embeddings (TAE) that are extracted based on the encoder outputs and CTC alignments. TAE can be obtained simultaneously without recurrent operations, leading to a parallel generation of output sequences. In addition, an error-based alignment sampling method is proposed to reduce alignment mismatch between the training and inference. CASS-NAT achieves ~20x speed up during inference without significant performance degradation compared to AT. We also propose a CASS-NAT variant (UniEnc-CASSNAT) that consists of only an encoder module. Together with the proposed multi-pass CTC training and iterative decoding, UniEnc-CASSNAT can perform as well as CASS-NAT with fewer model parameters.

Beyond these two challenges, and in order to facilitate the development of better child ASR, we build the first child ASR benchmark for the research community. The benchmark includes comparisons of widely-used techniques for ASR such as data augmentation, parameter efficient finetuning (PEFT), self-supervised (HuBERT and WavLM) and supervised models (such as Whisper). All codes developed in this dissertation will be available.

With the recent push of Automatic Speech Recognition (ASR) capabilities to mobile devices, the user's voice is now recorded in environments with a potentially high level of background noise. To reduce the sensitivity of ASR performance to these distortions, techniques have been proposed that preprocess the speech waveforms to remove noise effects while preserving discriminative speech information. At the expense of increased complexity, recent algorithms have significantly improved recognition accuracy but remain far from human performance in highly noisy environments.

With a concern for both complexity and performance, this thesis investigated ways to reduce the corruptive effect of noise by directly weighting the power-spectrum (SMF_pow) or log-spectrum (SMF_log) of speech by a mask whose values are within [0,1] and are indexed on the local relative prominence of speech and noise energy. Additional contributions include a low-complexity approach to mask estimation and the use of spectral flooring for matching the dynamic range of clean and noisy spectra. These two techniques are evaluated on two standard noisy ASR databases: the Aurora-2 connected digits recognition task with 11 words, and the Aurora-4 continuous speech recognition task with 5000 words.

On the Aurora-2 task, the SMF_log algorithm leads to state-of-the-art performance, with a limited complexity compared to existing techniques. The _pow technique, however, results in many insertions that we attribute to the rather weak language model present in the Aurora-2 setup. On the Aurora-4 task, both algorithms show significant improvements over the un-enhanced baselines. In particular, word-accuracies obtained with _pow approach those of a state-of-the-art front-end algorithm, on half of the noise types. Yet, the performances are heavily noise dependent, suggesting that the proposed technique is effective only given a good initial mask estimation.

This study confirms the potential of techniques that are based on direct spectrum masking, and proposes a framework for doing so. Future work will need to consider more elaborate mask estimation techniques to further improve on the performance.