UC Santa Barbara

The fields of visual prosthetics and virtual reality (VR) are intersecting in exciting ways. Designed to restore a rudimentary form of vision to people living with profound blindness, visual prostheses electrically stimulate surviving cells in the visual pathway to evoke visual percepts. Like VR headsets, these devices commonly use a head-mounted camera to capture visual data, updating the view as the user shifts position. Despite the growing use of VR headsets to simulate what people “see” using visual prostheses, most previous simulations lack biological realism or do not consider the way a prosthesis user would use head and eye movements to sample the scene.To address these challenges, I developed BionicVisionXR, an open-source VR toolbox for simulated prosthetic vision that uses a neurophysiologically inspired and psychophysically validated computational model to allow sighted participants to ‘see through the eyes’ of a prosthesis user. First, to demonstrate its utility, I systematically evaluated the effect of clinically reported perceptual distortions on performance in letter recognition and immersive obstacle avoidance tasks. Second, I enriched our simulations with gaze contingency and temporal effects to capture often neglected simulation parameters that may affect the quality of vision provided by existing devices. Third, to guide the development of next-generation devices, I propose a way of decomposing the scene into meaningful parts using edge detection and semantic segmentation. My results demonstrate the importance of choosing an appropriate level of immersion and phosphene model complexity. Furthermore, using a combination of computational modeling and behavioral testing, I was able to identify electrode stimulation strategies that may improve the quality of vision provided by retinal implants. This work has the potential to 1) further our understanding of the qualitative experience associated with different visual prosthetics, 2) provide realistic expectations of prosthetic performance for patients, doctors, manufacturers, and regulatory bodies, 3) accelerate the prototyping of new devices that may one day restore useful vision to people living with profound blindness.