Department of Computer Science

Visual cortical prostheses offer a promising path to sight restoration, but current systems elicit crude, variable percepts and rely on manual electrode-by-electrode calibration that does not scale. These limitations reflect a deeper challenge: electrical microstimulation evokes nonlinear, state-dependent population responses in the human visual cortex, complicating the link between stimulation and perception. Here, we present a deep learning framework that leverages a bidirectional cortical implant to causally shape stimulation-evoked population activity in the human visual cortex. The framework, trained on trial-resolved neural recordings, supports two complementary control strategies: a learned inverse network for real-time stimulation synthesis and a gradient-based optimizer for precise targeting. Both outperform conventional methods, achieve targets at lower stimulation currents, and elicit more consistent perception. Achievable responses lie on the intrinsic low-dimensional manifold of cortical activity, and recorded population activity predicts reported percepts substantially better than stimulation parameters alone. Together, these results provide a population-level foundation for linking microstimulation, cortical activity, and perception in the human visual system.

Visual systems evolved to extract behaviorally relevant information while animals move through and interact with their world. Such ecological vision differs fundamentally from standard laboratory paradigms in many key aspects, making this a much harder problem for the brain to solve, and for the neuroscientist to study. However, emerging technologies and experimental approaches have enabled investigation of visual computations under these ecological conditions. These approaches are particularly powerful in the mouse, combining well-developed genetic tools, high-throughput recordings, and quantifiable ethological tasks. Here we review computations that are engaged in ecological contexts, including active sensing, motion processing, scene analysis, distance estimation, and spatial perception. We delineate experimental approaches that engage these computations and synthesize current understanding of their neural implementations based on mouse research. These studies reveal how ecological vision engages distinct processing strategies and novel neural circuitry, while highlighting the vast territory that remains unexplored in understanding real-world visual computation.

Abstract Freely moving mice are generally thought to redirect gaze mainly through head-coupled movements, with eye movements stabilizing the retinal image or resetting eye position. This reflexive view leaves unresolved whether gaze shifts also include active coordination modes. Here we show that a single gaze-shift class resolves into multiple distinct head-eye coordination motifs. Using high-resolution head and eye tracking in freely moving mice, we identified four reproducible motifs: Head-before-Eye (HbE), Head-with-Eye (HwE), Head-Dominant (HD), and Eye-Dominant (ED). The motifs differed in head-eye timing, locomotor context, and pre-onset visual-behavioral structure. Their short-timescale sequential organization was non-random: HD formed alternating side-to-side head movements, whereas HwE-HbE-ED formed a directed transition chain preserved within each movement direction. Simultaneous neural recordings further revealed motif-dependent responses in primary visual cortex (V1) and superficial superior colliculus (sSC), with stronger sSC than V1 modulation for the three motifs distinguished by pre-onset visual-behavioral structure (all but HbE). HwE showed the clearest signature of active orienting, combining near-synchronous head-eye onset, strong relevance to visual-behavioral features, and the largest pre-onset and earliest peri-onset responses in sSC. These results expand a unitary view of gaze shifts in freely moving mice into a diverse, structured repertoire of coordination motifs, and identify HwE as a candidate active gaze-shift mode during natural behavior.

Central and peripheral vision loss, caused by conditions such as age-related macular degeneration and retinitis pigmentosa, disrupt visual processing in distinct ways, yet their impact on real-world scene perception remains poorly understood. Here, we used a real-time, gaze-contingent simulation to examine how central vision loss and peripheral vision loss alter eye movements and scene understanding. Sighted participants (n = 32, five males) viewed 120 real-world scenes (50% social interaction, 50% neutral) under one- or three-saccade constraints and described each scene; description quality was quantified via semantic similarity to ground-truth responses. Peripheral vision loss observers produced significantly less informative descriptions than both central vision loss and control participants, particularly for social interaction scenes, suggesting that peripheral vision is critical for rapid extraction of scene semantics. In contrast, central vision loss primarily disrupted oculomotor behavior, including increased saccade amplitudes, delayed saccade initiation, and reduced intersubject fixation consistency. Description quality was not predicted by fixation similarity to controls but by fixations to labeled humans and critical objects, underscoring the role of semantically informative sampling for real-world scenes that include people. These results reveal a dissociation between perceptual and oculomotor consequences of vision loss and highlight the importance of peripheral input for real-world scene understanding.

Purpose

Methods

Results

Conclusions

Translational relevance

Objective.Machine learning (ML) models are increasingly used to generate electrical stimulation patterns in neuroprosthetic devices such as visual prostheses. While these models promise precise and personalized control, they also introduce new safety risks when model outputs are delivered directly to neural tissue. We propose a systematic, quantitative approach to detect and characterize unsafe stimulation patterns in ML-driven neurostimulation systems.Approach.We adapt an automated software testing technique known as coverage-guided fuzzing to the domain of neural stimulation. Here, fuzzing performs stress testing by perturbing model inputs and tracking whether resulting stimulation violates biophysical limits on charge density, instantaneous current, or electrode co-activation. The framework treats encoders as black boxes and steers exploration with coverage metrics that quantify how broadly test cases span the space of possible outputs and violation types.Main results.Applied to deep stimulus encoders for the retina and cortex, the method systematically reveals diverse stimulation regimes that exceed established safety limits. Two violation-output coverage metrics identify the highest number and diversity of unsafe outputs, enabling interpretable comparisons across architectures and training strategies.Significance.Violation-focused fuzzing reframes safety assessment as an empirical, reproducible process. By transforming safety from a training heuristic into a measurable property of the deployed model, it establishes a foundation for evidence-based benchmarking, regulatory readiness, and ethical assurance in next-generation neural interfaces.

Purpose: Visual function testing in retinal prosthesis users relies on repetitive psychophysical tasks that are cognitively demanding and fatiguing. Gamification may increase engagement, but its effects on perceptual performance in implanted users remain unclear. Methods: Three Argus II users completed circle localization and motion direction discrimination in clinical and gamified versions. Visual stimuli, trial structure, and response requirements were matched within each participant; gamified versions added scoring, background music, and affectively framed end-of-trial auditory feedback. Difficulty and response format were calibrated to individual abilities (8AFC for two participants; 4AFC restricted to cardinal directions for one participant). Results: Gamification improved accuracy and reduced angular error in localization but did not improve motion discrimination. Effects were task-dependent and varied across participants, with reduced precision in the gamified motion task for one user. Participants preferred gamified localization and reported higher enjoyment and sustained attention; responses to gamified motion were mixed. Conclusions: Gamification can influence measured performance and user experience in prosthetic vision testing, but benefits are not universal and depend on task demands and cognitive load, indicating that engagement can affect outcomes in tests often treated as objective. Translational relevance: Personalized, engagement-aware gamified tools with adaptive difficulty may improve the usability and scalability of prosthetic vision assessment and rehabilitation, including at-home training.

A key limitation shared by both electronic and optogenetic sight recovery technologies is that they cause simultaneous rather than complementary firing within on- and off-center cells. Here, using "virtual patients"-sighted individuals viewing distorted input-we examine whether gamified training improves the ability to compensate for distortions in neuronal population coding. We measured perceptual learning using dichoptic input, filtered so that regions of the image that produced on-center responses in one eye produced off-center responses in the other eye. The Non-Gaming control group carried out an object discrimination task over five sessions using this filtered input. The Gaming group carried out an additional 25 hours of gamified training using a similarly filtered variant of the video game Fruit Ninja. Both groups showed improvements over time in the object discrimination task. However, there was no significant transfer of learning from the "Fruit Ninja" task to the object discrimination task. The lack of transfer of learning from video game training to object recognition suggests that gamification-based rehabilitation for sight recovery technologies may have limited utility and may be most effective when targeted on learning specific visual tasks.

Objective:Spatial scheduling of electrode activation ("rastering") is essential for safely operating high-density retinal implants, yet its perceptual consequences remain poorly understood. This study systematically evaluates the impact of raster patterns, or spatial arrangements of sequential electrode activation, on performance and perceived difficulty in simulated prosthetic vision (SPV). By addressing this gap, we aimed to identify patterns that optimize functional vision in retinal implants. Approach:Sighted participants completed letter recognition and motion discrimination tasks under four raster patterns (horizontal, vertical, checkerboard, and random) using an immersive SPV system. The simulations emulated epiretinal implant perception and employed psychophysically validated models of electrode activation, phosphene appearance, nonlinear spatial summation, and temporal dynamics, ensuring realistic representation of prosthetic vision. Performance accuracy and self-reported difficulty were analyzed to assess the effects of raster patterning. Main results:The checkerboard pattern consistently outperformed other raster patterns, yielding significantly higher accuracy and lower difficulty ratings across both tasks. The horizontal and vertical patterns introduced biases aligned with apparent motion artifacts, while the checkerboard minimized such effects. Random patterns resulted in the lowest performance, underscoring the importance of structured activation. Notably, checkerboard matched performance in the "No Raster" condition, despite conforming to groupwise safety constraints. Significance:This is the first quantitative, task-based evaluation of raster patterns in SPV. Checkerboard-style scheduling enhances perceptual clarity without increasing computational load, offering a low-overhead, clinically relevant strategy for improving usability in next-generation retinal prostheses.