Haptic Augmented Reality to monitor human arm's stiffness in rehabilitation

Augmented Reality (AR) is a live, direct or indirect, view of a physical, real-world environment whose elements are overlaid by virtual, computer generated objects. In this paper, AR is combined with haptics in order to observe human arm's stiffness. A haptic, hand-held device is used to measure the human arm's impedance. While a computer vision system tracks and records the position of the hand, a computer screen displays the impedance diagrams superimposed on the hand in a real-time video feed. The visual augmentation is also performed using a video projector that project's the diagrams on the hand as it moves.


INTRODUCTION
Stroke represents a major health concern for the American public, ranking as the third leading cause of death in the United States, and a leading cause of disability [1].Each year in the U.S. alone over 600,000 people survive a stroke, while similar figures exist in other countries.Approximately 80% of acute stroke survivors lose arm and hand movement skills [2].Stroke patients with movement impairments are required to have intensive physical therapy sessions after the initial injury occurs.Due to high expense of having an always-present therapist and problem of frequent commuting to clinic, homebased rehabilitation systems are in demand.
We have developed a low-cost haptic augmented reality rehab system that allows individual with stroke to practice their arm and hand movements without intervention of a therapist.The proposed system provides the current market with a new platform which saves considerable time, effort, and money.The system offers objective, quantitative assessment.

II. RELATED WORK
The research interest in applying AR in Rehabilitation Engineering (RE) and Assistive Technology (AT) fields is growing.This field of research is inter and multi-disciplinary that combines technologies of computer vision, haptics, sensors, etc.Many AT and RE systems have been developed in the last decade, and "it has been proven that AT and RE systems can greatly improve the quality of life of the people with disabilities" [3].
"Rehabilitation engineering applications are specially designed to recover certain impaired functions of individuals with disabilities" [3].Since our focus in this paper is haptic AR rehab systems, in the following, we discuss the state-of-the-art research that has been carried out in this domain.
Among the AR rehab systems, Alamri et al. [4] proposed a framework that takes advantages of the AR-based rehabilitation processes with a 2-D web camera and fiducial markers.The system supported the training of daily activity, though, due to use of several fiducial markers, the setup can be complex in the absence of therapist.GenVirtual, developed by Correa et al. [5], [6], was an AR game musical, that provided a natural, fingertip/toe tip-based interaction.Patients could use this system to practice their motor function coordination.However, patients with low mobility might not have fully leveraged the system.Luo et al. [7], [8] presented an environment for stroke survivors to have their hand opening function trained.To process the repetitive task of grasp-andrelease, they integrated AR with assistive technology.One problem with this system was though the equipments that patient has to put on before training which implies the necessity of therapist's intervention at some point.
Among the rehab systems, those which measure the mechanical impedance of the arm are of related to our work.The mechanical impedance of a musculoskeletal system is regulated by the Central Nervous System to control joints with redundant (agonists and antagonists) muscles.Hogan [9], [10] and Hondori [11] showed that the antagonist muscle's coactivation is meant to generate mechanical impedance which is necessary to perform manual and bimanual tasks.Burdet et al. [12] showed that human subjects adapt and control dynamically unstable tasks by optimizing mechanical impedance.Not just unstable dynamics but also stable tasks involve agonist and antagonist muscle's co-contraction as Darainy et al. [13] reported that the EMG patterns during learning a dynamically stable task shows a considerable share of antagonist co-activation.Considering the inherent signal dependant noise, higher muscular activations lead to more instability and instability lead to more co-activation of muscles; this creates a positive feedback in the system.Wolpert et al showed that the signal dependent noise determines motor planning [14] and the central nervous system's strategy to cope with this kind of noise is impedance control [15].In summary to the above-mentioned work, the antagonist muscle's cocontraction, that determines a joint's impedance, is seen in stable and unstable tasks; the co-contraction also causes spasticity of the arms which is a common problem in stroke patients.
Compared to the above platforms, in our system, augmented reality fuses with haptics to leverage the benefits of both.AR provides immersive experience for users which help objective quantification of the training process as well as motivation for massed practice.On the other hand, haptics gives natural feeling of interacting with objects which balances the experience of virtual and real-world objects on patients.

A. Experiment Setup
Our experimental setup comprises of two parts: computer vision and haptic.The computer vision part includes: a conventional PC, a low cost webcam, and a projector, whereas the haptic part includes a smart mug [16][17][18].Patient sits in front of a table that serves as a platform to practice hand and arm movements.While the patient is moving the smart mug from point A to point B, the camera monitors and records the motion, reporting any change in its position.Considering the input data obtained from the camera (position data) as well as the smart mug (impedance data), the system generates appropriate impedance graphs that are projected on patient's hand using the projector (Fig. 1) as well as displayed on the laptop screen (Fig. 2) in real time [19].The system has been designed in a way that is easy to setup at home by the patients, making it comfortable to use without therapist's presence.

B. Measuring Mechanical Impedance
Human limb's stiffness is usually measured using a robot that applies perturbation to hand, recording force and displacement to calculate ratio of them [9], [20][21][22][23][24].However, in this work, a mug-shape actuator, equipped with an inertia sensor, is customized for measuring wrist impedance [18].The device's design is very convenient to hold since it has the shape and weight of an ordinary mug (Fig. 3).Impedance highly depends on posture where it is also a function of perturbation direction.In one posture, the major axis of the ellipse (maximum impedance at that posture) shows the direction in which the hand is most stable while the minor axis (minimum impedance) attributes to the least stable direction.

C. AR System
As mentioned above, we obtain impedance in form of elliptical diagrams.The next step is to augment these ellipses on the patient's hand using color marker.We developed computer vision algorithms to locate and track the color marker.Our augmented reality system is combination of the tracking algorithm and the real-time overlaying of ellipses on the registered position of the patient's hand.
IV. EXPERIMENTAL DATA Fig. 4 shows the x and y positions of the mug during a time span of 30 seconds which consists of several reaching trials.Fig. 5 shows the x and y positions during a single reaching.First derivative of the position signals in Fig. 5 with respect to time, gives the x and velocities.The velocities are shown in Fig. 6.The signals were then fitted by a 4 th order polynomial using Matlab.The fitted curves can be seen in Fig. 7.
Once the position of the mug (or marker) is registered, the elliptical diagram of the impedance is added to the video feed and displayed on the computer screen.A sample of the added impedance diagram is shown in Fig. 8.To make the system more immersive, a projector is used for augmenting the impedance diagram on the actual mug.A sample of the augmentation is shown in Fig 9.The immersive user experience along with potential ease of use, motivation promoting massed practice, and objective quantitative assessment are obvious advantages to the fusion of a haptic AR rehab system.
In this paper, a haptic sensor that measures impedance of human arm is used.The haptic device applies perturbation to measure postural stiffness.Since the haptics measurements are based on local coordinate of the device, a computer vision system locates and tracks position and orientation of the haptic device.The experimental results for reaching task performed by a healthy human subject showed that the system can measure the postural impedance and augment the ellipses on the subject's hand in real time.In the future, the system is planned to be added to the smart intake gesture monitoring system for tele-rehab [25].

Figure 1 .
Figure 1.The subject moves the mug while the camera tracks it and the projector augments the impedance graph (ellipse) on the hand

Figure 2 .
Figure 2. The subject moves the mug while the camera tracks it and the screen displays the impedance graphs of the hand in real-time

Figure 3 .
Figure 3.The haptic device held by the subject's hand

Figure 4 .Figure 5 .Figure 6 .
Figure 4. Position of the mug which is tracked by the computer vision system over time period of 30 sec which involved several reaching tasks in different directions; y position is the solid and x position is the dashed line.

Figure 7 .
Figure 7.A 4 th order polynomials is fitted on velocities in a single reaching task; y velocity is the solid and x velocity is the dashed line.The bell-shaped speed profile is obvious.

Figure 8 .
Figure 8. Hand-held haptic device with a color marker on the top.The ellipse is augmented on the marker in the real time video feed seen on the screen.

Figure 9 .
Figure 9.The device with a color marker on the top.The ellipse is augmented on the actual marker in real time by a projector.
Corresponding author, Maryam Khademi, is with Donald Bren School of Informatics and Computer Science at the University of California, Irvine CA 92617.Correspondences shall be sent to the email provided above.HosseinMousaviHondori is with the University of California Irvine, CA 92617.Correspondences shall be sent to the email provided above or to hoss0001@e.ntu.edu.sg.Cristina V. Lopes is with Donald Bren School of Informatics and Computer Science at the University of California Irvine, CA 92617.Lucy Dodakian and Steve Cramer are with the University of California Irvine.978-1-4673-1666-8/12/$31.00 ©2012 IEEE 2012 IEEE EMBS International Conference on Biomedical Engineering and Sciences | Langkawi | 17th -19th December 2012