UC Davis

Human movements and perception of self-generated actions require the integration of multisensory information with motor commands to guide predictive control and error correction. Among the sensory channels that support movement, vision and haptics play distinct yet complementary roles. Vision provides broad spatial context and predictive information for planning and monitoring action, while haptic signals including tactile sensation and proprioception provide near-immediate, localized feedback about contact with external objects and body configuration. This integration is flexible and supports robust performance in the face of uncertainty: when one sensory modality becomes unreliable or unavailable, the nervous system can shift reliance toward alternative signals to better support state estimation and movement correction to maintain effective control under changing sensory and motor conditions. As sensory information becomes more informative or reliable, control can also shift from deliberate, high-cognitive load control to fluent, largely automatic performance as learning strengthens internal models and improves predictive control. However, it remains unclear how multisensory integration supports the perception of one’s actions when sensory information is altered, reduced, or re-encoded through sensory substitution or augmentation, both in everyday human behavior and during interaction with robotic systems.This dissertation examines how visual and haptic information jointly shape human motor control, and how these processes extend to interactions with assistive devices and robotic systems. It begins with an examination of how sensory feedback, when coupled with motor actions, influences perceptual integration and embodiment, such that a robotic prosthetic hand and its actions are experienced as part of the user’s own body (Chapter 2). This chapter demonstrates that the degree to which an upper limb prosthesis is experienced as part of the body depends on sensorimotor contingencies, showing how changes in sensory feedback and action consequences reshape body representation and embodiment.Next, this dissertation examines tactile sensory substitution for navigation (Chapter 3). The chapter shows that tactile cues can deliver spatial information when vision is reduced or absent, and it explains how this information is used to construct spatial understanding during movement. Following this, multisensory integration through sensory substitution or augmentation extends to multi-limb coordination, a core for seamless interaction with external control systems (Chapter 4). Upon investigation, participants can adapt to extracting relevant patterns from haptic feedback, integrating them with vision, and progressively reducing reliance on visual monitoring while maintaining stable and effective motor performance.Finally, teleoperated robotic control places perception and action under constrained and indirect feedback, where operators often rely heavily on vision to infer the remote task state and predict the consequences of their actions (Chapter 5). This chapter introduces a wearable haptic interface that conveys task relevant robotic information and provides a foundational work that adding haptic feedback redistributes information away from vision, shape control strategies, and support skill acquisition during teleoperation.Across these studies, the dissertation positions various forms of haptic feedback as complementary information channels that can be added to, or substituted for, vision. This dissertation shows how humans integrate augmentative haptic information with vision and motor commands to support bodily self-representation, understanding of the surrounding environment, and coordinated movement when interacting with external systems (from the self to external systems). Together, these findings bridge fundamental understanding of multisensory integration and motor control and clarify how augmentative haptic feedback can be structured usability, learning, and reliable performance in assistive and teleoperated robots.