An assistive tabletop keyboard for stroke rehabilitation

We propose a tabletop keyboard that assists stroke patients in using computers. Using computers for purposes such as paying bills, managing bank accounts, sending emails, etc., which all include typing, is part of Activities of Daily Living (ADL) that stroke patients wish to recover. To date, stroke rehabilitation research has greatly focused on using computer-assisted technology for rehabilitation. However, working with computers as a skill that patients need to recover has been neglected. The conventional human computer interfaces are mouse and keyboard. Using keyboard stays the main challenge for hemiplegic stroke patients because typing is usually a bimanual task. Therefore, we propose an assistive tabletop keyboard which is not only a novel computer interface that is specially designed to facilitate patient-computer interaction but also a rehab medium through which patients practice the desired arm/hand functions.


Introduction
Stroke is a leading cause of serious long-term disability in adults. More than 795,000 people in the United States suffer from a stroke each year [3], [4]; this costs the country an estimated $38.6 billion that includes the cost of healthcare services and missed days of work. Computer-assisted technology has a key role in enhancing the traditional physical and occupational therapy, improving healthcare service, and decreasing the associated costs [5], [6].

Background and Related Work
Because this study stresses rehabilitation of fine motor control of fingers, here we discuss some of the related work in the same domain. Interestingly, a good number of related work requires the patient to don a customized glove to register finger movements while playing a computer game. For example, Carmeli et al. [7] introduced HandTutor, a glove-based treatment system whose exercises are based on repetitive and intensive active flexion and extension movement of finger(s) and the wrist. In a similar attempt, Reinkensmeyer et al. [8] designed MusicGlove to motivate and quantify finger movement by using functional grips to play music. Also, Lord et al. developed a virtual-reality environment for training finger individuation in stroke survivors with chronic hand impairment; the users play an ergonomic 5-key virtual piano while receiving either assistance or resistance from a pneumatically actuated glove. In a non-glove-based study, Correa et al. [9], [10] developed GenVirtual, an musical game that provided a natural, fingertip/toe interaction.
Despite focus of the discussed work, i.e., using computer-assisted technology for rehabilitation, we aim at building a patient-computer interface that directly assists patients with their computer skills while also helping them retain their fine motor controls. Training patients with an interface (i.e., keyboard) to help them with a natural ADL (i.e., typing) may have a greater functional gain than with an unfamiliar interface (e.g., glove).

Methodology
The assistive keyboard is similar to a regular keyboard in appearance. The main difference is being adaptive and interactive. To use the assistive keyboard, the patient sits at a table on which virtual keyboard is projected. The patient's hand movements are detected and tracked by the camera, while (s)he is looking on the table and interacting with the virtual keyboard ( Fig.  1). Our system has the potential to be used locally, in a clinical environment, as well as remotely, in the patient's home where it can remain accessible by a therapist through a telerehabilitation system.

Setup
The platform includes a conventional computer, a depth camera, and a micro projector. Our setup components are very light and portable (Fig. 2); the setup is suitable to send to patients' home as part of a telerehab package.
We tend to give patients the same experience that they used to have with a regular keyboard; patients have the keyboard positioned in a close distance/attached to a monitor. Therefore, we use Creative Intel Depth Sensor. It is a small, light-weight, USB-powered camera optimized for close-range interactivity. Designed for ease of setup and portability, it includes an HD webcam, depth sensor and built-in dual-array microphones for capturing and recognizing voice, gestures and images. Using Intel depth sensor, we are able to incorporate close-range (15-40cm) hand

Motivation
Our specific aims are to investigate whether the assistive keyboard can return the patient's confidence in using computers while in parallel the patient achieves better gains over their fine motor controls. Our motivation to investigate these hypotheses has shaped during one of our preliminary studies [1], [2] with stroke patients. Reviewing our questionnaires, we noticed that many patients wish to recover the skill of typing. For example, some of them mentioned how frustrated they get while sending an email. One of the patients mentioned that he tried to use speech recognition tools but they were not flexible to delete the mistyped words. He also said that the speech recognition tools are to compensate the disfunctionality but he wanted to retain his right hand and not to make it obsolete. Based on the major need of our patients to get back to computers, we decided to design and develop an assistive keyboard for them.

Posters
ITS'13, October 6-9, 2013, St. Andrews, UK tracking. We also use AAXA's P4-X Pico Projector. The P4-X's high-brightness optical engine is capable of delivering 80 lumens in DC power mode and 60 lumens in battery mode. Operating time is 75 minutes on a built-in battery or unlimited when plugged into a wall outlet.

Interactive Interface
The assistive keyboard is an interactive interface. It provides patients with dynamic audio-visual feedbacks to guide them through different tasks as well as encourage them not to stop practicing. For example, the patient may enable the audio to hear which letter has been pressed. The patient may also wish to use any combination of auto-correction/auto-completion modes while typing. The assistive keyboard can also alleviate typing for the patients by showing only possible characters that can follow up a certain typed letter (Fig. 3). This way, the keyboard will only show a subset of keys and disable the others. This feature can reduce typing error.

Adaptive Interface
One of the key advantages of the assistive keyboard is its adaptability to patients' specific needs. Due to the wide spectrum of motor or sensory deficits in stroke patients [11], adaptability is a major issue in patientcomputer interface design. For example, if the patient has inaccurate targeting, the system can project larger keys. As the patient gets progress, the system can adjust the size of keys by reducing their area as well as the space between them until the keys reach the standard size and distance. Another example is leveraging Visual Error Augmentation and Reduction to enhance rehabilitative learning (Fig. 4). Wei et al. [12] showed that "error augmentation may 'wake up' an inattentive nervous system and trigger the recovery process by supplying heightened, magnified sensory feedback about a persons' motor deficit."

Involved Hand/Arm Movements
Using the assistive keyboard requires various hand/arm movements ranging from goal-directed reaching, maintaining a hand posture, wrist abduction & adduction, wrist supination & pronation, and finger individuation with proper range, speed, and smoothness of movements (Fig. 5). Using these parameters, we can measure major performance factors that are useful to assess a patient's progress across therapy weeks.

Hand Tracking
A core component of the system is hand tracking. We have developed several vision algorithms to track the hands using depth images. We plan to further improve our algorithms so that they can recognize the individual fingers too. Using finger recognition and tracking techniques, our system will be able to monitor which keys are pressed by which fingers. This way, we can control if the patient is using the affected hand/fingers or trying to compensate by using the healthy ones. Also we will be able to adjust our task/games so that they only give patients score if they use the right finger(s) to type a certain letter. This method can monitor finger individuation (Fig. 6) with a good accuracy. Also because typing is a bimanual task, we hope to encourage patients to use the affected hand in coordination with the healthy one.

Keyboard Games
Along with the implementation of the assistive keyboard, we have included desktop version of several engaging games, which involve typing, in our platform. These games include "Guess the Brand", "Guess the Flag", "Guess the Country" that are currently 4.5-5 star ranked games in iTunes and Google Play [13]. In these games, a picture representing a brand, logo, flag, or country is displayed and the player has to type corresponding letters to complete the word in a given time period (e.g., Fig. 7).

Conclusion and Future Work
The assistive keyboard is one step towards extending the well-established field of Human-Computer Interaction to Patient-Computer Interaction where the patient in our case is a stroke survivor with motor and cognitive deficits. We anticipate that stroke community will embrace the idea of assistive/therapeutic keyboard because working with computers is a central ADL that stroke patients wish to recover (based on our patients' self-report) while conventional rehabilitation may not achieve this purpose. We have already designed a between-subject study to investigate the effectiveness of our assistive keyboard compared with traditional therapy on stroke patients. The results will be published in near future.