Skip to main content
eScholarship
Open Access Publications from the University of California

Scholarly Works (10 results)

  • Thesis
  • Peer Reviewed
UC Merced Electronic Theses and Dissertations (2024)

Auditory attention to memory is often overlooked compared to the visual domain. Our aim is to expand research within the auditory domain using a retro-cue paradigm. This paradigm allows us to investigate the attentional effects on auditory short-term memory representations through the strengthening and/or focusing of these memory representations. This study involved tasking participants to use their short-term auditory memory, and a visual retro-cue to detect a change in a temporal auditory feature, using amplitude modulated (AM) sounds. The visual retro-cue was either informative or uninformative. An informative retro-cue is a stimulus that effectively reduces the working memory load of the participant in accordance with the memory probe task, thereby increasing accuracy, d’, and/or response time on the memory probe task. The uninformative retro-cue is a stimulus that does not provide insight into the upcoming memory probe task and is used as a comparative baseline for the informative retro-cue. We hypothesized, based on past auditory research, that relative to uninformative retro-cues, the informative retro-cues would improve accuracy and response time, especially when detecting smaller changes in the probe’s amplitude modulated AM rate relative to its original AM rate. However, due to various limitations, our hypothesis was unable to be verified. Future directions solving these limitations will hopefully be taken in the future. We found trends in the data that, when the limitations are addressed, may lead to significance once the study is revisited.

    Cover page: Using a Retro-Cue Paradigm to Probe the Temporal Precision of Auditory Memory Representations
    • Article
    • Peer Reviewed
    UC Merced Previously Published Works (2021)
    A debate over the past decade has focused on the so-called bilingual advantage-the idea that bilingual and multilingual individuals have enhanced domain-general executive functions, relative to monolinguals, due to competition-induced monitoring of both processing and representation from the task-irrelevant language(s). In this commentary, we consider a recent study by Pot, Keijzer, and de Bot (2018), which focused on the relationship between individual differences in language usage and performance on an executive function task among multilingual older adults. We discuss their approach and findings in light of a more general movement towards embracing complexity in this domain of research, including individuals' sociocultural context and position in the lifespan. The field increasingly considers interactions between bilingualism/multilingualism and cognition, employing measures of language use well beyond the early dichotomous perspectives on language background. Moreover, new measures of bilingualism and analytical approaches are helping researchers interrogate the complexities of specific processing issues. Indeed, our review of the bilingualism/multilingualism literature confirms the increased appreciation researchers have for the range of factors-beyond whether someone speaks one, two, or more languages-that impact specific cognitive processes. Here, we highlight some of the most salient of these, and incorporate suggestions for a way forward that likewise encompasses neural perspectives on the topic.
      Cover page: Characterizing Bilingual Effects on Cognition: The Search for Meaningful Individual Differences
      • Thesis
      • Peer Reviewed
      UC Merced Electronic Theses and Dissertations (2024)

      With the rise of bilinguals globally, extensive research has been conducted to understand how bilingualism affects cognitive functions differently from monolingualism, with significant implications for bilingual education, healthcare, and other fields. Specific to language processing, differences have been identified between second language (L2) speakers and monolinguals, particularly in contexts such as speech perception in noise and executive control. Despite the advances in this field of language processing, there remains a critical gap in understanding how bilinguals' phonological awareness of their first language (L1) and L2 influences their phonetic encoding and categorization. In this dissertation, I examine how various top-down and bottom-up factors modulate phonemic encoding in bilinguals, both behaviorally and neurophysiologically. The first chapter investigates how ambiguous audiovisual situations result in a default phonetic perception, providing a foundation for the second chapter. The second chapter explores how visual information influences the phonetic encoding of specific phonemes by Spanish-English bilinguals. Lastly, the third chapter addresses the influence of selective attention, lexicosemantic encoding, and low-level sound encoding on bilingual phonetic encoding, focusing on how bilinguals respond to phonemic restoration. This experiment is conducted using electroencephalography (EEG) to provide insights into the neurophysiological mechanisms underlying these processes. By integrating behavioral and neurophysiological approaches, this dissertation aims to uncover the complex mechanisms underlying phonetic encoding in bilinguals, contributing to a comprehensive understanding of bilingual cognitive processing and its broader implications.

        Cover page: How top-down and bottom-up factors modulate bilingual phonemic encodingCreative Commons 'BY' version 4.0 license
        • Article
        • Peer Reviewed
        UC Merced Previously Published Works (2020)
        There is growing interest in characterizing the neural mechanisms underlying the interactions between attention and memory. Current theories posit that reflective attention to memory representations generally involves a fronto-parietal attentional control network. The present study aimed to test this idea by manipulating how a particular short-term memory (STM) representation is accessed, that is, based on its input sensory modality or semantic category, during functional magnetic resonance imaging (fMRI). Human participants performed a novel variant of the retro-cue paradigm, in which they were presented with both auditory and visual non-verbal stimuli followed by Modality, Semantic, or Uninformative retro-cues. Modality and, to a lesser extent, Semantic retro-cues facilitated response time relative to Uninformative retro-cues. The univariate and multivariate pattern analyses (MVPAs) of fMRI time-series revealed three key findings. First, the posterior parietal cortex (PPC), including portions of the intraparietal sulcus (IPS) and ventral angular gyrus (AG), had activation patterns that spatially overlapped for both modality-based and semantic-based reflective attention. Second, considering both the univariate and multivariate analyses, Semantic retro-cues were associated with a left-lateralized fronto-parietal network. Finally, the experimental design enabled us to examine how dividing attention cross-modally within STM modulates the brain regions involved in reflective attention. This analysis revealed that univariate activation within bilateral portions of the PPC increased when participants simultaneously attended both auditory and visual memory representations. Therefore, prefrontal and parietal regions are flexibly recruited during reflective attention, depending on the representational feature used to selectively access STM representations.
          Cover page: Orienting Attention to Short-Term Memory Representations via Sensory Modality and Semantic Category Retro-Cues
          • Article
          • Peer Reviewed
          UC Merced Previously Published Works (2021)
          The McGurk illusion occurs when listeners hear an illusory percept (i.e., da), resulting from mismatched pairings of audiovisual (AV) speech stimuli (i.e., auditory/ba/paired with visual/ga/). Hearing a third percept-distinct from both the auditory and visual input-has been used as evidence of AV fusion. We examined whether the McGurk illusion is instead driven by visual dominance, whereby the third percept, e.g., da, represents a default percept for visemes with an ambiguous place of articulation (POA), like/ga/. Participants watched videos of a talker uttering various consonant vowels (CVs) with (AV) and without (V-only) audios of/ba/. Individuals transcribed the CV they saw (V-only) or heard (AV). In the V-only condition, individuals predominantly saw da/ta when viewing CVs with indiscernible POAs. Likewise, in the AV condition, upon perceiving an illusion, they predominantly heard da/ta for CVs with indiscernible POAs. The illusion was stronger in individuals who exhibited weak/ba/auditory encoding (examined using a control auditory-only task). In Experiment2, we attempted to replicate these findings using stimuli recorded from a different talker. The V-only results were not replicated, but again individuals predominately heard da/ta/tha as an illusory percept for various AV combinations, and the illusion was stronger in individuals who exhibited weak/ba/auditory encoding. These results demonstrate that when visual CVs with indiscernible POAs are paired with a weakly encoded auditory/ba/, listeners default to hearing da/ta/tha-thus, tempering the AV fusion account, and favoring a default mechanism triggered when both AV stimuli are ambiguous.
            Cover page: Rethinking the Mechanisms Underlying the McGurk Illusion.Creative Commons 'BY' version 4.0 license
            • Article
            • Peer Reviewed
            UC Merced Previously Published Works (2023)
            Recent studies have questioned past conclusions regarding the mechanisms of the McGurk illusion, especially how McGurk susceptibility might inform our understanding of audiovisual (AV) integration. We previously proposed that the McGurk illusion is likely attributable to a default mechanism, whereby either the visual system, auditory system, or both default to specific phonemes-those implicated in the McGurk illusion. We hypothesized that the default mechanism occurs because visual stimuli with an indiscernible place of articulation (like those traditionally used in the McGurk illusion) lead to an ambiguous perceptual environment and thus a failure in AV integration. In the current study, we tested the default hypothesis as it pertains to the auditory system. Participants performed two tasks. One task was a typical McGurk illusion task, in which individuals listened to auditory-/ba/ paired with visual-/ga/ and judged what they heard. The second task was an auditory-only task, in which individuals transcribed trisyllabic words with a phoneme replaced by silence. We found that individuals' transcription of missing phonemes often defaulted to '/d/t/th/', the same phonemes often experienced during the McGurk illusion. Importantly, individuals' default rate was positively correlated with their McGurk rate. We conclude that the McGurk illusion arises when people fail to integrate visual percepts with auditory percepts, due to visual ambiguity, thus leading the auditory system to default to phonemes often implicated in the McGurk illusion.
              Cover page: The McGurk Illusion: A Default Mechanism of the Auditory SystemCreative Commons 'BY' version 4.0 license
              • Article
              • Peer Reviewed
              UC Merced Previously Published Works (2022)
              Previous work addressing the influence of audition on visual perception has mainly been assessed using non-speech stimuli. Herein, we introduce the Audiovisual Time-Flow Illusion in spoken language, underscoring the role of audition in multisensory processing. When brief pauses were inserted into or brief portions were removed from an acoustic speech stream, individuals perceived the corresponding visual speech as "pausing" or "skipping", respectively-even though the visual stimulus was intact. When the stimulus manipulation was reversed-brief pauses were inserted into, or brief portions were removed from the visual speech stream-individuals failed to perceive the illusion in the corresponding intact auditory stream. Our findings demonstrate that in the context of spoken language, people continually realign the pace of their visual perception based on that of the auditory input. In short, the auditory modality sets the pace of the visual modality during audiovisual speech processing.
                Cover page: Audition controls the flow of visual time during multisensory perception.
                • Article
                • Peer Reviewed
                UC Merced Previously Published Works (2023)
                Traditionally, pitch variation in a sound stream has been integral to music identity. We attempt to expand music's definition, by demonstrating that the neural code for musicality is independent of pitch encoding. That is, pitchless sound streams can still induce music-like perception and a neurophysiological hierarchy similar to pitched melodies. Previous work reported that neural processing of sounds with no-pitch, fixed-pitch, and irregular-pitch (melodic) patterns, exhibits a right-lateralized hierarchical shift, with pitchless sounds favorably processed in Heschl's gyrus (HG), ascending laterally to nonprimary auditory areas for fixed-pitch and even more laterally for melodic patterns. The objective of this EEG study was to assess whether sound encoding maintains a similar hierarchical profile when musical perception is driven by timbre irregularities in the absence of pitch changes. Individuals listened to repetitions of three musical and three nonmusical sound-streams. The nonmusical streams were comprised of seven 200-ms segments of white, pink, or brown noise, separated by silent gaps. Musical streams were created similarly, but with all three noise types combined in a unique order within each stream to induce timbre variations and music-like perception. Subjects classified the sound streams as musical or nonmusical. Musical processing exhibited right dominant α power enhancement, followed by a lateralized increase in θ phase-locking and spectral power. The θ phase-locking was stronger in musicians than in nonmusicians. The lateralization of activity suggests higher-level auditory processing. Our findings validate the existence of a hierarchical shift, traditionally observed with pitched-melodic perception, underscoring that musicality can be achieved with timbre irregularities alone.NEW & NOTEWORTHY EEG induced by streams of pitchless noise segments varying in timbre were classified as music-like and exhibited a right-lateralized hierarchy in processing similar to pitched melodic processing. This study provides evidence that the neural-code of musicality is independent of pitch encoding. The results have implications for understanding music processing in individuals with degraded pitch perception, such as in cochlear-implant listeners, as well as the role of nonpitched sounds in the induction of music-like perceptual states.
                  Cover page: Neurophysiological time course of timbre-induced music-like perceptionCreative Commons 'BY-NC-SA' version 4.0 license
                  • Article
                  • Peer Reviewed
                  UC Davis Previously Published Works (2022)

                  Purpose

                  This research examined the expression of cortical auditory evoked potentials in a cohort of children who received cochlear implants (CIs) for treatment of congenital deafness (n = 28) and typically hearing controls (n = 28).

                  Method

                  We make use of a novel electroencephalography paradigm that permits the assessment of auditory responses to ambiently presented speech and evaluates the contributions of concurrent visual stimulation on this activity.

                  Results

                  Our findings show group differences in the expression of auditory sensory and perceptual event-related potential components occurring in 80- to 200-ms and 200- to 300-ms time windows, with reductions in amplitude and a greater latency difference for CI-using children. Relative to typically hearing children, current source density analysis showed muted responses to concurrent visual stimulation in CI-using children, suggesting less cortical specialization and/or reduced responsiveness to auditory information that limits the detection of the interaction between sensory systems.

                  Conclusion

                  These findings indicate that even in the face of early interventions, CI-using children may exhibit disruptions in the development of auditory and multisensory processing.
                    Cover page: Electrophysiological Examination of Ambient Speech Processing in Children With Cochlear Implants.
                    • Article
                    • Peer Reviewed
                    UC Merced Previously Published Works (2024)
                    Research on action-based timing has shed light on the temporal dynamics of sensorimotor coordination. This study investigates the neural mechanisms underlying action-based timing, particularly during finger-tapping tasks involving synchronized and syncopated patterns. Twelve healthy participants completed a continuation task, alternating between tapping in time with an auditory metronome (pacing) and continuing without it (continuation). Electroencephalography data were collected to explore how neural activity changes across these coordination modes and phases. We applied deep learning methods to classify single-trial electroencephalography data and predict behavioral timing conditions. Results showed significant classification accuracy for distinguishing between pacing and continuation phases, particularly during the presence of auditory cues, emphasizing the role of auditory input in motor timing. However, when auditory components were removed from the electroencephalography data, the differentiation between phases became inconclusive. Mean accuracy asynchrony, a measure of timing error, emerged as a superior predictor of performance variability compared to inter-response interval. These findings highlight the importance of auditory cues in modulating motor timing behaviors and present the challenges of isolating motor activation in the absence of auditory stimuli. Our study offers new insights into the neural dynamics of motor timing and demonstrates the utility of deep learning in analyzing single-trial electroencephalography data.
                      Cover page: Investigating the role of auditory cues in modulating motor timing: insights from EEG and deep learning
                      Top