Free-hand interaction with leap motion controller for stroke rehabilitation

In recent years, the field of Human-Computer Interaction (HCI) has been advanced with many technologies, however, most are limited to healthy users. In this paper, we leveraged the technology of free-hand interaction to rehabilitate patients with stroke. We modified the game of Fruit Ninja to use Leap Motion controller's hand tracking data for stroke patients with arm and hand weakness to practice their finger individuation. In a pilot study, we recruited 14 patients with chronic stroke to play the game using natural interaction. Their Fruit Ninja (FN) scores show high correlation with the standard clinical assessment scores such as Fugl-Meyer (FMA) and Box-and-Blocks Test (BBT) scores. This finding suggests that our free-hand Fruit Ninja's score is a good indicator of the patient's hand function and therefore will be informative if used in their rehabilitation.

While using free-hand interaction systems for an ablebodied user may be considered a luxury, it might serve as a great assistive technology for patients with physical disability.
One of the patient communities who may benefit largely from hand interaction are individuals with stroke. Stroke causes hemiplegia which affects motor functions in one side of the body; typically, patients lose part or full control of one of their hands. Considering that stroke is a leading cause of serious long-term disability in adults with more than 795,000 people in the United States each year [6], [7], helping this community to retain their hand motor control, could have a significant impact on their quality of life.

Background and Related Work
The human-computer interaction community has made several attempts to enhance stroke rehabilitation with HCI technologies. Hallam et al. [8] proposed an interactive glove to foster acceptance of partners after one of them is affected by stroke and help them reunite. Boulanger et al. [9] used Microsoft Surface's hand position as input of a tabletop game environment for stroke rehabilitation. The study involved exercises such as curling and uncurling fingers as well as a wrist flexion and extension of the hand back and forth about the wrist. Another example of fine motor rehab systems is PointAssist [10] which is a mouse interface that adapts to a user's level of impairment in reaching and clicking. The system could improve click success rates of individuals with fine motor impairments. Also [11] evaluated a novel click assistance technique to help users with motor impairments to click more accurately using a mouse. More specific studies focusing on finger rehabilitation include but are not limited to HandTutor [12], a glove-based treatment system which provides intensive flexion and extension movement of finger(s) and the wrist. Also MusicGlove by Reinkensmeyer et al. [13] motivates finger movements by pressing fingers against each other to play GuitarHero.
The above research shows example of touch, mouse, glove-based technologies that have been applied to stroke rehabilitation. Yet, using free-hand interaction as the most intuitive option to stroke rehabilitation has to be investigated.

A Gaming System for Finger Individuation
We combined the game of Fruit Ninja with the Leap sensor, using open source JavaScript version of Fruit Ninja [14]. The desktop version of the game is supposed to be played with mouse events such as click and drag. Since our focus is finger individuation, we modified the game so that mouse events can be replaced by hand movements. Using the hand tracking information of the Leap sensor, we were able to trigger equivalent mouse events. We have also previously used Fruit Ninja in other stroke rehabilitation settings [15].
The setup includes a PC/notebook and a Leap sensor that is plugged in via its USB (Fig. 1). The game runs on a browser (e.g. Google Chrome) while the hand is monitored using our java API code which is written on top of the Leap sensor's SDK.

Study
To determine the feasibility of free-hand interaction using Leap Motion controller for stroke rehabilitation exercises, we conducted a pilot study with 14 patients with stroke. The participants were asked to play our free-hand Fruit Ninja game. We also simplified the original Fruit Ninja so that patients with stroke with different levels of hand deficits are able to play the game for one minute.

Game
To avoid turning the rehabilitation exercises into boring and repetitive tasks, exercises are often gamified. For this purpose, we used Fruit Ninja, a top-ranked game in iTunes and Google Play [16]. Fruit Ninja is not only an engaging game but also requires goal directed hand movements, which makes it appropriate for hand rehabilitation. In the original version, as the game starts, different kinds of fruits appear on the screen while the player has to slice them as fast as possible with mouse click and drag events (in the desktop version) versus touch and drag events (in the mobile version) (See Fig. 2). If the player misses slicing three fruits, the game is over. Also, in the course of the game, several bombs appear on the screen time to time that the player has to avoid; otherwise the game is over.
We modified the Fruit Ninja game so that the patients with stroke were able to play. We changed the gameover restriction by removing the three missing fruits condition and disabling the bombs; instead we introduced a time constraint of 1 minute. This gave all the patients a similar game condition which was required for conducting a fair study and comparing their results.

Experimental Procedure
Four participants had difficulty with finger movements; we strapped a stick (one of a pair of chop sticks) to the dorsum of their hand (Fig. 3); this was to enable them to play the game using their wrist movements (e.g., wrist supination and pronation). Since our experiment was part of a bigger study for which these four patients passed the inclusion criteria, we also accepted them to try the Fruit Ninja game. However, we took note of their special condition and did not compare their results with those of the rest of the patients. After providing informed consent, the patients were briefed on how to play the game. They also had a warm-up round to get accustomed to the game. All the patients were asked to use their stroke-affected hand. They had three, oneminute rounds of playing the game while they were given break time in-between to rest. At the end, we asked some Likert questions on a scale of 1-4 to learn about the patients' overall satisfaction. These questions included: (i) whether the patient found the game engaging on a scale of 1 (not engaging) to 4 (very engaging); (ii) whether the game addressed a need (yes/no); and (iii) whether the patient would use the game at home on a scale of 1 (wouldn't use) to 4 (would use).   Fig. 4 shows the game scores of our participants against their FM and BBT scores. To make an accurate comparison between patients' scores, we removed the score of four out of 14 patients who used the stick instead of their fingers. As shown in Fig. 4 In response to whether the game was engaging for the patients, 11 out of 14 subjects gave 4/4 (very engaging). Also 10 of them agreed that the game was addressing a need, i.e., the game provided practicing the movements that can be generalized to daily functions. In response to the question whether the patients would be willing to play this game if provided at home, 12 of them picked 4/4 (would use) while the other two participants gave 1/4 (wouldn't use). Two of the patients (P6-male and P1-female) who enjoyed playing the Fruit Ninja commented that the game provided hand-eye coordination which is very important in activities of daily living (ADLs). Also, three participants (P8-female, P13-male, and P7-male) mentioned that the game was not very responsive in terms of tracking their hand.

Discussion
Following up the tracking issue mentioned in previous section, we revisited the data of those particular patients (P8-female, P13-male, and P7-male) and found out that they had difficulty individuating their index finger to make a pointing gesture. Instead, they played the game with all the fingers extended. The Leap sensor assigns temporary ID's (1-5) to different fingers, for example, moving from frame n to n+1, the index finger may or may not be assigned to ID = 2. This creates a subtle confusion while showing the trace of slice on the screen (the trace may appear with jittering). Note that this event may happen in few percents of the frames (around 1%).
To solve this problem, we propose a 3D model of the hand that includes specific ID's for each finger and contains all skeletal parameters of the hand. Given that Leap's data is limited to fingertip position and direction vector, palm center position, normal, and direction vector, we used inverse kinematics to extract finger joint angles.
The human hand has 27 Degrees of Freedom (DoF) [19] where each joint contributes to 1-3 DoF. The wrist has 3 Cartesian DoF (x, y, and z) and 3 rotational DoF (roll, pitch, and yaw). Finger joints include the metacarpophalangeal or MP, proximal interphalangeal or PIP, and distal interphalangeal or DIP joints. The thumb has 5 DoF including 1 at the DIP, 2 at the PIP, and 2 at the MP. Each other finger (i.e., index, middle, ring, and fifth) has 4 DoF, including 1 at the DIP, 1 at the PIP, and 2 at the MP.
Of these 27 DoF, Leap SDK directly provides only the 3 from wrist rotations but it provides enough data to compute the rest. We used inverse kinematics to calculate the MP, PIP, and DIP angles by going through fingertip position and direction vector of each finger to the palm's center. Fig. 6 shows the completed hand skeleton based on the Leap's data shown in Fig. 5. The hand skeletal model not only increases the robustness, but also enables us to record and reconstruct hand movements in a realistic way for medical assessments. Besides, the model can be used for simulating patients' hand movements in a virtual environment (i.e., games or computerized therapeutic exercises) where patients manipulate objects.

Conclusion
The HCI advents have not been fully explored in the field of patient-computer interaction. In this paper, we used the cutting-edge technology of free-hand interaction with Leap Motion controller for stroke rehabilitation. We modified the Fruit Ninja game to use the Leap sensor's hand tracking data. The combination was prepared for patients with stroke to practice their fine motor control. We conducted a pilot study with 14 patients with stroke to evaluate feasibility of using this system for rehabilitation of upper extremity. The results demonstrated significant correlations between scores generated from the Fruit Ninja game and standard clinical outcome measures, such as the Fugl-Meyer Arm assessment and Box-and-Blocks Test. The qualitative evaluation of the system was also proved successful. To make the system more accurate and responsive, we also proposed a kinematic model of the hand. Using this model allows us to incorporate 3D parameters of the hand in tracking which in turn makes tracking more robust.