Skip to main content
eScholarship
Open Access Publications from the University of California

Scholarly Works (28 results)

  • Article
  • Peer Reviewed
UC San Francisco Previously Published Works (2018)
OBJECTIVE: Recent research has characterized the anatomical and functional basis of speech perception in the human auditory cortex. These advances have made it possible to decode speech information from activity in brain regions like the superior temporal gyrus, but no published work has demonstrated this ability in real-time, which is necessary for neuroprosthetic brain-computer interfaces. APPROACH: Here, we introduce a real-time neural speech recognition (rtNSR) software package, which was used to classify spoken input from high-resolution electrocorticography signals in real-time. We tested the system with two human subjects implanted with electrode arrays over the lateral brain surface. Subjects listened to multiple repetitions of ten sentences, and rtNSR classified what was heard in real-time from neural activity patterns using direct sentence-level and HMM-based phoneme-level classification schemes. MAIN RESULTS: We observed single-trial sentence classification accuracies of [Formula: see text] or higher for each subject with less than 7 minutes of training data, demonstrating the ability of rtNSR to use cortical recordings to perform accurate real-time speech decoding in a limited vocabulary setting. SIGNIFICANCE: Further development and testing of the package with different speech paradigms could influence the design of future speech neuroprosthetic applications.
    Cover page: Real-time classification of auditory sentences using evoked cortical activity in humans.
    • Article
    • Peer Reviewed
    UC San Francisco Previously Published Works (2014)
    Speech perception requires rapid integration of acoustic input with context-dependent knowledge. Recent methodological advances have allowed researchers to identify underlying information representations in primary and secondary auditory cortex and to examine how context modulates these representations. We review recent studies that focus on contextual modulations of neural activity in the superior temporal gyrus (STG), a major hub for spectrotemporal encoding. Recent findings suggest a highly interactive flow of information processing through the auditory ventral stream, including influences of higher-level linguistic and metalinguistic knowledge, even within individual areas. Such mechanisms may give rise to more abstract representations, such as those for words. We discuss the importance of characterizing representations of context-dependent and dynamic patterns of neural activity in the approach to speech perception research.
      Cover page: Dynamic speech representations in the human temporal lobe
      • Article
      • Peer Reviewed
      UC San Francisco Previously Published Works (2016)

      Objective

      The superior temporal gyrus (STG) and neighboring brain regions play a key role in human language processing. Previous studies have attempted to reconstruct speech information from brain activity in the STG, but few of them incorporate the probabilistic framework and engineering methodology used in modern speech recognition systems. In this work, we describe the initial efforts toward the design of a neural speech recognition (NSR) system that performs continuous phoneme recognition on English stimuli with arbitrary vocabulary sizes using the high gamma band power of local field potentials in the STG and neighboring cortical areas obtained via electrocorticography.

      Approach

      The system implements a Viterbi decoder that incorporates phoneme likelihood estimates from a linear discriminant analysis model and transition probabilities from an n-gram phonemic language model. Grid searches were used in an attempt to determine optimal parameterizations of the feature vectors and Viterbi decoder.

      Main results

      The performance of the system was significantly improved by using spatiotemporal representations of the neural activity (as opposed to purely spatial representations) and by including language modeling and Viterbi decoding in the NSR system.

      Significance

      These results emphasize the importance of modeling the temporal dynamics of neural responses when analyzing their variations with respect to varying stimuli and demonstrate that speech recognition techniques can be successfully leveraged when decoding speech from neural signals. Guided by the results detailed in this work, further development of the NSR system could have applications in the fields of automatic speech recognition and neural prosthetics.
        Cover page: Neural speech recognition: continuous phoneme decoding using spatiotemporal representations of human cortical activity
        • Article
        • Peer Reviewed
        UC San Francisco Previously Published Works (2019)
        The human superior temporal gyrus (STG) is critical for extracting meaningful linguistic features from speech input. Local neural populations are tuned to acoustic-phonetic features of all consonants and vowels and to dynamic cues for intonational pitch. These populations are embedded throughout broader functional zones that are sensitive to amplitude-based temporal cues. Beyond speech features, STG representations are strongly modulated by learned knowledge and perceptual goals. Currently, a major challenge is to understand how these features are integrated across space and time in the brain during natural speech comprehension. We present a theory that temporally recurrent connections within STG generate context-dependent phonological representations, spanning longer temporal sequences relevant for coherent percepts of syllables, words, and phrases.
          Cover page: The Encoding of Speech Sounds in the Superior Temporal Gyrus
          • Thesis
          • Peer Reviewed
          UC San Diego Electronic Theses and Dissertations (2011)

          A growing majority of the world's population is able to speak and understand more than one language, yet most of our knowledge of how the brain processes words comes from monolingual speakers of a small number of languages. Furthermore, most studies of both monolingual and bilingual individuals have utilized brain imaging methods that provide either excellent spatial or temporal resolution, but not both. In aggregate, these studies have demonstrated that information is encoded in both local and regional brain networks across time, so it is therefore necessary to understand the specific spatiotemporal dynamics of these processes.

          This dissertation seeks to advance our knowledge of the various stages of word processing across languages in bilinguals and across the various modalities in which language occurs. Using a multimodal imaging approach that combines the temporal resolution of magnetoencephalography (MEG) with the spatial resolution of magnetic resonance imaging (MRI), we focus on an early word form encoding stage and a later lexico-semantic stage in both the visual and auditory modalities.

          In two studies with bilingual participants, we show that the less proficient language (regardless of whether it was learned first or second) recruits an extended network of brain regions beginning during the earliest stages of word encoding (~150 ms for written words and ~100 ms for auditory words) and continuing through lexico-semantic processing (~400 ms).

          In a third study, we investigate the properties of a newly discovered neural response that occurs at ~100 ms to auditory words. This response represents the encoding of acoustic information into a phonemic code, which can then be passed onto the classical lexico-semantic brain areas (and the extended network in less proficient languages) for integrating the word into the surrounding semantic context.

          These studies help clarify the importance of proficiency and experience in evaluating how bilinguals process each of their languages, which provides a unique perspective on the general process of language acquisition. Furthermore, understanding the spatiotemporal dynamics of word encoding in both the visual and auditory modalities allows us to identify fundamental neural mechanisms that are modality- and stimulus- independent.

          • 2 supplemental videos
          Cover page: Neural Stages of Word Encoding Across Languages, Proficiency Levels, and Modalities
          • Article
          • Peer Reviewed
          UC Berkeley Previously Published Works (2024)
          Melody is a core component of music in which discrete pitches are serially arranged to convey emotion and meaning. Perception varies along several pitch-based dimensions: (i) the absolute pitch of notes, (ii) the difference in pitch between successive notes, and (iii) the statistical expectation of each note given prior context. How the brain represents these dimensions and whether their encoding is specialized for music remains unknown. We recorded high-density neurophysiological activity directly from the human auditory cortex while participants listened to Western musical phrases. Pitch, pitch-change, and expectation were selectively encoded at different cortical sites, indicating a spatial map for representing distinct melodic dimensions. The same participants listened to spoken English, and we compared responses to music and speech. Cortical sites selective for music encoded expectation, while sites that encoded pitch and pitch-change in music used the same neural code to represent equivalent properties of speech. Findings reveal how the perception of melody recruits both music-specific and general-purpose sound representations.
            Cover page: Encoding of melody in the human auditory cortex.
            • Article
            • Peer Reviewed
            UC San Francisco Previously Published Works (2018)
            Auditory speech comprehension is the result of neural computations that occur in a broad network that includes the temporal lobe auditory cortex and the left inferior frontal cortex. It remains unclear how representations in this network differentially contribute to speech comprehension. Here, we recorded high-density direct cortical activity during a sine-wave speech (SWS) listening task to examine detailed neural speech representations when the exact same acoustic input is comprehended versus not comprehended. Listeners heard SWS sentences (pre-exposure), followed by clear versions of the same sentences, which revealed the content of the sounds (exposure), and then the same SWS sentences again (post-exposure). Across all three task phases, high-gamma neural activity in the superior temporal gyrus was similar, distinguishing different words based on bottom-up acoustic features. In contrast, frontal regions showed a more pronounced and sudden increase in activity only when the input was comprehended, which corresponded with stronger representational separability among spatiotemporal activity patterns evoked by different words. We observed this effect only in participants who were not able to comprehend the stimuli during the pre-exposure phase, indicating a relationship between frontal high-gamma activity and speech understanding. Together, these results demonstrate that both frontal and temporal cortical networks are involved in spoken language understanding, and that under certain listening conditions, frontal regions are involved in discriminating speech sounds.
              Cover page: Neural correlates of sine-wave speech intelligibility in human frontal and temporal cortex
              • Article
              • Peer Reviewed
              UC San Francisco Previously Published Works (2018)
              Intracranial electrical recordings and stimulation of neurosurgical patients have been central to the advancement of human neuroscience. The use of these methods has rapidly expanded over the last decade due to theoretical and technical advances, as well as the growing number of neurosurgical patients undergoing functional procedures for indications such as epilepsy, tumor resection, and movement disorders. These methods pose the potential for ethical conflict, as they involve basic neuroscientific research utilizing invasive procedures in human patients undergoing treatment for neurological illnesses. This review addresses technical aspects, clinical contexts, and issues of ethical concern, utilizing a framework that is informed by, but also departs from, existing bioethical literature on matters in clinical research. We conclude with proposals for improving informed consent processes to address potential problems specific to intracranial electrophysiology research, a general schema for scrutinizing research-related risk associated with different methods, and a call for the development of consensus to ensure continuing scientific progress alongside crucial patient protections in this promising area of human neuroscience.
                • Article
                • Peer Reviewed
                UC San Francisco Previously Published Works (2020)
                The fluent production of a signed language requires exquisite coordination of sensory, motor, and cognitive processes. Similar to speech production, language produced with the hands by fluent signers appears effortless but reflects the precise coordination of both large-scale and local cortical networks. The organization and representational structure of sensorimotor features underlying sign language phonology in these networks remains unknown. Here, we present a unique case study of high-density electrocorticography (ECoG) recordings from the cortical surface of profoundly deaf signer during awake craniotomy. While neural activity was recorded from sensorimotor cortex, the participant produced a large variety of movements in linguistic and transitional movement contexts. We found that at both single electrode and neural population levels, high-gamma activity reflected tuning for particular hand, arm, and face movements, which were organized along dimensions that are relevant for phonology in sign language. Decoding of manual articulatory features revealed a clear functional organization and population dynamics for these highly practiced movements. Furthermore, neural activity clearly differentiated linguistic and transitional movements, demonstrating encoding of language-relevant articulatory features. These results provide a novel and unique view of the fine-scale dynamics of complex and meaningful sensorimotor actions.
                  Cover page: Cortical Encoding of Manual Articulatory and Linguistic Features in American Sign Language
                  • Article
                  • Peer Reviewed
                  UC San Francisco Previously Published Works (2020)
                  In speech, listeners extract continuously-varying spectrotemporal cues from the acoustic signal to perceive discrete phonetic categories. Spectral cues are spatially encoded in the amplitude of responses in phonetically-tuned neural populations in auditory cortex. It remains unknown whether similar neurophysiological mechanisms encode temporal cues like voice-onset time (VOT), which distinguishes sounds like /b/ and/p/. We used direct brain recordings in humans to investigate the neural encoding of temporal speech cues with a VOT continuum from /ba/ to /pa/. We found that distinct neural populations respond preferentially to VOTs from one phonetic category, and are also sensitive to sub-phonetic VOT differences within a population's preferred category. In a simple neural network model, simulated populations tuned to detect either temporal gaps or coincidences between spectral cues captured encoding patterns observed in real neural data. These results demonstrate that a spatial/amplitude neural code underlies the cortical representation of both spectral and temporal speech cues.
                    Cover page: Transformation of a temporal speech cue to a spatial neural code in human auditory cortex
                    Top