Robots capable of robust, real-time recognition of human intent during manipulationtasks could be used to enhance human-robot collaboration for innumerable applications. Eye gaze-based control interfaces offer a non-invasive way to infer intent and reduce the cognitive burden on operators of complex robots. Eye gaze is traditionally used for “gaze triggering” (GT) in which staring at an object, or sequence of objects, triggers pre-programmed robotic movements. Our long-term objective is to leverage eye gaze as an intuitive way to infer human intent, advance action recognition for shared autonomy control, and enable seamless human-robot collaboration not yet possible with state-of-the-art gaze-based methods. In Study #1, we identified features from 3D gaze behavior for use by machine learning classifiers of action recognition. We investigated gaze behavior and gaze-object interactions as participants performed the bimanual activity of preparing a powdered drink. We generated 3D gaze saliency maps and used characteristic gaze object sequences to demonstrate an action recognition algorithm. In Study #2, we introduced a classifier for recognizing action primitives, which we defined as triplets having a verb, “target object,” and “hand object.” Using novel 3D gaze-related features, a recurrent neural network was trained to recognize a verb and target object. The gaze object angle and its rate of change enabled accurate recognition and a reduction in the observational latency of the classifier. Using a non-specific approach to indexing objects, we demonstrated modest generalizability of the classifier across activities. In Study #3, we introduced a neural network-based “action prediction” (AP) mode into a shared autonomy framework capable of 3D gaze reconstruction, real-time intent recognition, object localization, obstacle avoidance, and dynamic trajectory planning. Upon extracting gaze-related features, the AP model recognized, and often predicted, the operator’s intended action primitives. The AP control mode often preferred over a state-of-the-art GT mode, enabled more seamless human-robot collaboration. In summary, we developed machine learning-based action recognition methods using novel 3D gaze-related features to enhance the shared autonomy control of robot manipulators. Our methods can serve as a foundation for further enhancement with complementary sensory feedback such as computer vision and tactile sensing.

The unpredictability of unstructured environments poses a significant challenge to fully autonomous systems. A human in the loop can provide high-level decisions that leverage experience, context, and human creativity. Allowing humans to operate remotely through teleoperation improves flexibility and safety in hazardous conditions. Our work addresses human-in-the-loop teleoperation applications in assistive robotics and hazardous environments with granular materials. We demonstrate the critical role that haptic feedback plays in enabling humans and robots to operate in unstructured environments effectively.

In study #1, we developed a tactile-based proximity sensing framework for estimating the distance to buried objects in granular materials by exploiting granular media jamming. As a sensorized robot finger displaces granular material towards a buried object, jamming granular particles leads to increased fingertip contact forces. A tactile sensor effectively acts as a proximity sensor when moved through granular materials. Using our framework, the distance to a buried object can be estimated using solely haptic feedback.

In study #2, we developed a structured approach for estimating the orientation and diameter of cables buried in sand using a deformable, vision-based tactile sensor. Our orientation estimation method identifies the contact region of the cable by employing a proximity function that takes extracted image features as input. Our approach is compared with existing image feature extraction methods, including PCA and maximally stable extremal regions. Our framework demonstrates the effectiveness of tactile sensing when an object is partially or fully occluded in granular materials.

In study #3, we designed a telepresence system with real-time auditory, visual, and haptic feedback. The system comprises a modified Stretch mobile manipulator with additional controllable degrees of freedom in the wrist and fingers. The operator interface consists of a virtual reality headset and a haptic glove. The operator controls the remote robot using intuitive controls based on motion tracking. To convey tactile information, the operator receives electrical impulses to stimulate the afferent sensory nerves of each finger driven by tactile feedback from the robot hand.

In summary, we developed tactile-based systems and approaches to enhance human-in-the-loop teleoperation for applications in unstructured environments, such as in the home and in granular materials.

To execute manipulation tasks in unstructured environments, robots use computer vision and a priori information to locate and grasp objects of interest. However, once an object has been grasped, cameras cannot perceive tactile- or force-based information about finger-object interactions. To address this, tactile and proprioception data are used to develop novel methodologies that aid in robotic manipulation once an object has been grasped.

In the first study, a method was developed for the perception of tactile directionality using convolutional neural networks (CNNs). The deformation of a tactile sensor is used to perceive the direction of a tangential stimulus acting on the fingerpad. A primary CNN was used to estimate the direction of perturbations applied to a grasped object. A secondary CNN provided a measure of uncertainty through the use of confidence intervals. Our CNN models were able to perceive tactile directionality on par with humans, outperformed a state-of-the-art force estimator network, and was demonstrated in real-time.

In the second study, novel controllers were developed for model-free, tension-driven manipulation of deformable linear objects (DLOs) using force-based data. Prior works on DLO manipulation have focused on geometric or topological state and used complex modeling and computer vision approaches. In tasks such as wrapping a DLO around a structure, DLO tension needs to be carefully controlled. Such tension control cannot be achieved using vision alone once the DLO becomes taut. Two controllers were designed to regulate the tension of a DLO and precede traditional motion controllers. The controllers could be used for tasks in which maintaining DLO tension takes higher priority over exact DLO configuration. We evaluate and demonstrate the controllers in real-time on real robots for two different utilitarian tasks: circular wrapping around a horizontal post and figure-eight wrapping around a boat cleat.

In summary, methods were developed to effectively manipulate objects using tactile- and force-based information. The model-free nature of the approaches allows the techniques to be utilized without exact knowledge of object properties. Our methods that leverage tactile sensation and proprioception for object manipulation can serve as a foundation for further enhancement with complementary sensory feedback such as computer vision.

In Study #1, we develop a novel, data-driven method to estimate the kinematic parameters of multi-joint linkages such as the human hand. The method can identify up to two consecutive revolute joint axis orientations between connected rigid bodies. We introduce kinematic constraints into a Generative Topographic Mapping formulation in order to estimate the joint axis parameters. The method was evaluated using simulated motion and via motion capture and a physical 2-DOF mechanism modeled after the metacarpophalangeal joint of the human finger. Our method compares well against state-of-the-art kinematic parameter estimation techniques with regards to reliability and computational efficiency.

In Study #2, we introduce a sensor-embedded soft skin capable of multimodal sensing (contact force and two axes of shear force) in pressurized underwater environments up to 1000kPa. We embed liquid-metal strain gauges within a durable elastomeric skin that is molded around a solid finger core. We demonstrate that the sensor skin is capable of measuring forces up to 220N underwater and while subjected to a range of hydrostatic pressures. We determine that the performance of the sensor skin is unaffected by the submerged, pressurized environment.

In Study #3, we propose a set of design considerations for tactile sensor skins using embedded, microfluidic single-axis strain gauges for the purpose of estimating 3D forces and 1D torque about the skin's surface normal. By displacing shear force taxels such that their principal axes are offset from the point of contact, we are able to more accurately measure torque. We use an experimental testbed to apply force-torque loads to the sensor skin. We develop CNN-based models to evaluate the combined force-torque estimation performance of numerous taxel configurations and provide a detailed discussion of how performance relates to design choices.

In summary, we developed methods that improve sensing in harsh environments such as granular media and underwater. Kinematic and kinetic considerations during hand-object interaction were carefully integrated into the development of novel sensing hardware, data-driven estimation methods, and task-specific sensor design criteria. We advanced the state-of-the-art in tactile sensing using tactile sensor skins and improved the accuracy of hand pose estimation using low-cost motion tracking tools.

With the fast-paced proliferation of robots in our daily lives, it is especially important that the human experience is prioritized for human-robot interactions and automated systems. We investigated how human eye movements can be leveraged in real-time by robotic systems to enhance the experience of humans that interact with and/or observe semi-autonomous robots. Specifically, we studied spatiotemporal relationships between human eye gaze, human intent, and human-automation trust for activities of daily living.In Study #1, we identified features from 3D gaze behavior for use by machine learning classifiers for the purpose of action recognition. We investigated gaze behavior and gaze- object interactions as participants performed a bimanual activity of preparing a powdered drink. We generated 3D gaze saliency maps and used characteristic gaze object sequences to demonstrate an action recognition algorithm. In Study #2, we introduced a classifier for recognizing action primitives, which we defined as triplets having a verb, “target object,” and “hand object.” Using novel 3D gaze-related features, a recurrent neural network was trained to recognize a verb and target object. The gaze object angle and its rate of change enabled accurate recognition and a reduction in the observational latency of the classifier. Using a non-specific approach for indexing objects, we demonstrated potential generalizability of the classifier across activities. In Study #3, we evaluated subjective and objective measures of trust in human-automation interactions. We compared real-time physiological responses and trust levels (reported via joystick) to the state-of-the-art method of post-trial Likert surveys. Our results suggest that eye gaze features and heart rate are effective, nonintrusive metrics for real-time monitoring of human trust in automated systems. In summary, we developed machine learning-based action recognition methods using novel 3D gaze-related features, and we related uncertainty in robot competence to real- time measures of trust variation. Our work establishes a foundation for enhancing human- robot collaborative systems by leveraging eye tracking for intention recognition and trust calibration.