See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/266630214

Brain-Computer Interface Virtual Keyboard for Accessiblity

Article · June 2012

DOI: 10.2316/P.2012.772-036

CITATION READS
1 112

5 authors, including:

Jonathan Corley Dustin Heaton

University of West Georgia University of Alabama
17 PUBLICATIONS   100 CITATIONS    6 PUBLICATIONS   129 CITATIONS   

SEE PROFILE SEE PROFILE

Jeff Gray Jeff Carver

University of Alabama University of Alabama
356 PUBLICATIONS   3,401 CITATIONS    132 PUBLICATIONS   2,978 CITATIONS   

SEE PROFILE SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Aspect Oriented Modeling View project

Block Programming View project

All content following this page was uploaded by Jeff Gray on 30 December 2014.

The user has requested enhancement of the downloaded file.

 BRAIN-COMPUTER INTERFACE VIRTUAL KEYBOARD
FOR ACCESSIBILITY
Jonathan Corley, Dustin Heaton, Jeff Gray, Jeffrey C. Carver, Randy Smith
Department of Computer Science
University of Alabama
{corle001, dwheaton}@crimson.ua.edu {gray, carver, rsmith}@cs.ua.edu

ABSTRACT eMotiv headset detects sensorimotor signals, which are

This paper describes our experiences in building a virtual neurological signals related to motor control. This
keyboard implemented using a Brain-Computer Interface approach differs from many other implementations. For
(BCI) that interacts with the eMotiv EPOC Neural example, P300-based systems rely on a signal from the
Headset. The contribution of the work is an alternative brain that is generated as a response to outside stimuli [4].
input device for those who have a motor disability and are Systems that rely on outside stimuli force the selection
challenged by traditional input devices. The advantages of rate to be determined by the stimuli, as opposed to the
a virtual keyboard based on BCI are summarized and we user. The eMotiv headset does not rely on any outside
describe its design and implementation. We also present stimuli, which we believe will give users a greater sense
the results of a preliminary study that has suggested of control.
several improvements for enhancing the effectiveness of In the remainder of this paper, we present a novel
the virtual keyboard. system designed with the goal of producing a viable
alternative input device for those unable to use a physical
KEY WORDS keyboard. We will first identify a number of challenges
Brain-Computer Interface; BCI; Virtual Keyboard; that must be overcome for our system to be a useful
Accessibility; Accommodating People with Disabilities; keyboard alternative, and then we will outline how we
User Interface Development; Emotiv; End-User addressed those challenges. We will then present a
Development preliminary study that evaluates the current
implementation of our system. The study compares our
results to those obtained from several other similar BCI
1. Introduction
systems. Finally, we will discuss lessons learned from
both the user study and the design.
General computing skills continue to be an important
workforce need, and the inability to use such a common
instrument as the physical keyboard alienates certain 2. Design of a BCI Virtual Keyboard
individuals from a significant portion of the workforce,
and society in general. In order to provide an accessibility This section outlines some of the challenges encountered
option for those who are unable to use a physical and the design approaches that we considered while
keyboard (e.g., those with a motor disability), we have building our BCI virtual keyboard.
created a Brain-Computer Interface (BCI) Virtual
Keyboard. We believe a large majority of the population 2.1. Challenges in using a BCI Virtual Keyboard
that is unable to use a keyboard or mouse will be able to
use a BCI in a meaningful way. This belief is supported When considering the intended user base of the system
by other studies in the area of accessibility, such as (i.e., those with a motor disability), we identified the
Neuper et al. [5]. following challenges that guided our design:
We utilized the eMotiv EPOC neural headset 1. Not all users will be capable of producing each
(hereafter referred to as the eMotiv headset) [1] as our input signal with the same consistency;
physical input device for several reasons. The eMotiv 2. Not all users will be capable of producing the full
headset is a consumer-ready and relatively inexpensive range of input signals;
option for end-users. Use of consumer-ready systems has 3. The design should be reusable with different
been suggested to provide a practically portable solution languages;
for recording EEG signals [2]. In addition, if our virtual 4. User familiarity, or lack thereof, with keyboard
keyboard proves to be successful, Emotiv provides an layouts could affect user experience;
application store for developers using the eMotiv headset 5. The large number of inputs available on a modern
that would enable us to easily distribute our software. The keyboard could be overwhelming if presented to
the user all at once;
 6. The device we are using has a limited number of Robot class to send keyboard input from the virtual
available inputs that is greatly exceeded by the keyboard. The Robot class allows programmatic control
number of inputs available with a traditional of the mouse and keyboard (e.g., directing the mouse to a
keyboard. specific location of the screen or programmatically
generating keyboard button events).
In addition to the listed challenges and design The overall architecture of the design focuses on an
objectives, we considered the effect of requiring users to implementation of the Model-View-Controller design
train with the eMotiv headset. A new user will need to pattern. In this implementation, the model is the Virtual
spend an initial period of training time to configure the Keyboard, the view is the Graphical Display, and the
eMotiv headset reliably and accurately. This training time controller handles the mapping of headset inputs to
may be a burden to some users upon first use. However, functions we designed in the model. We believe this
we believe our target user base will have the motivation to architecture allows for flexibility in our implementation.
endure a period of training. Although we would like to This flexibility can be realized by altering the controller to
study the effect of varying training times in the future, we allow for different devices to utilize the same model and
currently are not certain about the optimal amount of display. Additionally, the display could be tailored to
training that will be required from users. Therefore, to various user expectations dependent on platform-specific
mediate the impact of this unknown factor, a user is only or societal conventions.
required to produce a subset of the full range of input During the initial design phase, we identified a
signals the eMotiv headset is capable of processing. In our number of challenges that were used to guide our design.
most recent interactions with experimental users, we We overcame several of these challenges through user
observed that a 10 minute training time was sufficient. We customization. First, in order to address Challenge 1 (i.e.,
plan to investigate this issue further in future works. that not all users will be capable of producing each signal
with the same consistency), we allowed the headset inputs
2.2. Details of the Design and Implementation to be mapped to functions based on an XML configuration
file. The input mapping allows a user to choose the most
This project was developed in Java using Eclipse. Java appropriate inputs for his or her ability (e.g., left- or right-
was chosen because of our past experience and the wink). To address Challenges 2 and 6, our system is
relative ease of designing a graphical user interface in designed to only require at least two distinct input signals.
Java. The eMotiv API is provided as a native C++ DLL. Each available input signal maps to one of the following
However, Emotiv provides a wrapper that utilizes the Java actions that are ordered by an initial assessment of
Native Access library to enable Java developers to utilize usefulness (most useful to least useful):
the eMotiv API. Additionally, the project uses the Java

Figure 1. Annotated Screen Shot of the BCI Virtual Keyboard

 1. Select 3.1. User Study Details and Experimental Context
2. Move Right
3. Move Out In order to evaluate the effectiveness of our system to
4. Move Down assist a user in entering text without a physical keyboard,
5. Move Left we performed a user study with seven participants. The
6. Move Up user tests were set up as follows:
1. We utilized the eMotiv EPOC Control Panel to
To realize this limited number of inputs, we created a familiarize each participant with the facial
model of the keyboard where the user navigates through feature recognition system (Expressiv).
the keys to make a selection. Because of Challenge 5, we 2. We configured a user profile for each participant,
needed to minimize the average number of movements including the training and sensitivity
required to navigate to each key. We accomplished this by adjustments, the input mappings, and the
utilizing a drill-down approach in which the user starts at keyboard layout.
a high-level and selects a set of keys and then navigates 3. We allowed each participant approximately 10
within this reduced set, selecting increasingly smaller sets minutes to become familiar with the keyboard
until the current set contains only a single key. At this layout and navigation scheme.
point, the user selects the desired key and the system 4. We tasked each participant with typing a
simulates the user pressing that key. In addition, each selection of text using the virtual keyboard
movement utilizes a wrapping implementation to allow utilizing only the Expressiv suite. Each
the ability to reach any key. Also, in order to address participant had 5 minutes to complete this task.
Challenges 3 and 4, we allow the keyboard layout to be 5. We utilized the eMotiv EPOC Control Panel to
mapped from an XML file. This allows the keyboard to be train each participant to perform one input from
tailored to the layout most comfortable to the user and to the mental state recognition system (Cognitiv).
utilize the appropriate keys for the language(s) of the user. Each subject was given approximately 10
A screenshot of the current graphical interface of the minutes to complete this training task.
system is shown in Figure 1, where the three levels of the 6. We updated each user profile based on the
drill-down approach are indicated: group, section, and feedback from steps 4 and 5.
key. In this architecture, a group contains sections and a 7. We allowed participants to familiarize
section contains keys. The user begins at the group level, themselves with utilizing the Cognitiv input in
selects a group, then selects a section from within that the context of the virtual keyboard.
group, and finally selects a particular key. 8. We tasked each participant with typing a
The eMotiv headset provides several recognition selection of text using the virtual keyboard
suites: the Expressiv and Cognitiv suites. The Expressiv utilizing both the Expressiv and Cognitiv suites.
suite detects neural signals related to facial feature Each participant had 5 minutes to complete this
movements (e.g., a blink). The Cognitiv suite allows the task.
user to train the system to recognize custom neural states. 9. We asked each participant to complete a brief
The Expressiv suite provides enough available input survey composed of the following questions that
signals to satisfy the needs of our virtual keyboard. were measured on a 5-point scale:
However, we recognized that not all users are capable of a. How would you rate the responsiveness of
utilizing the full suite. The Cognitiv suite provides the the keyboard without Cognitiv input?
freedom to define inputs that are tailored to an individual. b. How would you rate the responsiveness of
Unfortunately, as more custom inputs are added, the the keyboard with Cognitiv input?
difficulty for the user to reproduce a specific signal c. How would you rate the accuracy of the
increases significantly, as stated in the eMotiv SDK keyboard without Cognitiv input?
documentation [1]. Therefore, we allow the user to utilize d. How would you rate the accuracy of the
a combination of inputs from both suites in order to keyboard with Cognitiv input?
ensure that the capability of the full system is available to e. In a scenario involving daily use, assuming a
all users. standard keyboard is not available, how
comfortable would you be with using this
3. Empirical Evaluation of the Keyboard device?
f. In a scenario involving occasional use,
In this section, we present the design and results of an assuming a standard keyboard is not an
empirical user study that was used to evaluate the current option, how comfortable would you be with
implementation of our system. using this device?
g. The device met your expectations
h. Overall Satisfaction
 took the same participants 39 seconds to make each input
Average Average
Error Rate when using Cognitiv input. Additionally, the average error
User Inputs per seconds /
(%) rate when participants were not using Cognitiv input was
Selection selection
38.7%, which increased to 76.5% when participants
user1 75.11 84.50 100.00 performed the same task while using Cognitiv input. One
user2 30.97 27.20 90.00 participant was excluded from the analysis of the Cognitiv
input because their average selection time suggested that
user3 37.36 27.43 85.71 they were unable to control the system properly to provide
user5 67.84 16.50 100.00 meaningful input.
Participants rated the responsiveness when they were
user6 42.37 34.50 83.33
not using Cognitiv as 3.7. When Cognitiv input was
user7 37.76 46.20 0.00 added, the responsiveness rating dropped to 2.7.
Participants rated their comfort using the system in an
AVG. 48.57 39.39 76.51
occasional use scenario with an average of 2.3. In a daily
user4 312.46 332.00 100.00 use scenario, the average comfort rating dropped to 1.7.
Table 1. Results with Cognitiv Input The majority of users felt that the device met their
(user 4 excluded due to inability to use system) expectations, with all but one user rating this criterion a 3
or better. However, the average overall satisfaction was a
Avgerage Average 2.7, indicating the participants were slightly unsatisfied
Error Rate with the system. Details of the results of the user survey
User Inputs per seconds /
(%) can be found in Table 3.
Selection selection
user1 15.99 6.16 57.89
4. Related Works
user2 43.91 24.43 28.57
There are several BCI applications described in the
user3 18.30 13.13 6.25 literature. We found similar systems that provide context
and comparison with our own virtual keyboard. These
user4 22.73 9.29 71.43
previous results are summarized in Table 4 along with
user5 50.29 26.50 66.67 relevant system details.
After reviewing similar works, we found that the
user6 32.51 20.45 18.18 studies with the highest number of input selections
possible had between 25 and 40 available selections. Our
user7 16.64 10.89 22.22
system possesses 128 distinct inputs before considering
AVG. 28.62 15.83 38.75 key combinations. However, we believe this will not be a
significant hindrance from achieving similar results. The
Table 2. Results without Cognitiv Input
study performed by Sellers and Donchin demonstrated
that the technique they utilized (P300-based speller)
3.2. Results of our User Evaluation required extending the time per selection to an
unreasonable rate in order to improve the accuracy to an
As shown in Table 1 and Table 2, every measured acceptable level [6]. Thompson et al. presented an
criterion was improved significantly when using the evaluation of a plug-and-play BCI interface which
system without Cognitiv input (as compared to when included an evaluation of three P300-based spelling
using it with Cognitiv input). In particular, the average interfaces [7]. The three interfaces evaluated were
time it took the participants to make a selection when not BCI2000, DynaWrite, and Compass. They found that the
using Cognitiv input was 15.8 seconds. On average, it BCI2000 environment resulted in a 13% mean error rate
but had an 11% standard deviation. DynaWrite had a 13%
User/Question a b c d e f g h
1 3 2 2 3 2 4 4 4
2 4 4 3 1 1 2 3 3
3 3 1 3 1 2 2 3 2
4 4 3 4 3 1 1 1 2
5 4 1 3 1 2 2 3 2
6 4 5 3 1 2 3 3 3
Average 3.67 2.67 3.00 1.67 1.67 2.33 2.83 2.67
Table 3. Results from User Survey
 System Number of Available Error Selection Rate
Reference
Type Selections Rate (%) (seconds per selection)
Sellers and Donchin [6] P300 36 25 22.4
Sellers and Donchin [6] P300 36 8 106
Yue et al. [3] Sensorimotor 40 15 NA
Neuper et al. [5] Sensorimotor 26 15-30 ~60
Thompson, et al. [7] P300 36 11 3.5
Thompson, et al. [7] P300 36 13 3.5
Thompson, et al. [7] P300 36 13 3.5
Table 4. Related Works
mean error rate, but a 9% standard deviation, meaning the follow-up survey. Providing additional training time
that it was equal to BCI2000 in a worst-case scenario. has the potential to increase both the accuracy and
Compass again had a 13% mean error rate but only a 5% responsiveness, along with the user’s overall satisfaction.
standard deviation. The accuracy reported by Thompson Though this increase in training time may seem
et al. was obtained with a selection interval of 3.5s. detrimental to the eventual adoption of our system or a
The technique used by Yue et al. demonstrated a similar implementation, we believe the target user-base
comparable accuracy rate using a sensorimotor rhythms- will have the motivation to endure the training times,
based speller [3]. In addition, Neuper et al. presented a which are also limited to the first use of the keyboard.
study with accuracy rates similar to those proved by Yue Another significant factor was the lack of direction
given to the participants during the training process. The
et al., also using a sensorimotor rhythm-based speller. The
participants were introduced to the device and the virtual
study conducted by Neuper et al. is particularly
keyboard through basic descriptions of functionality. In
noteworthy, because the participants were patients
order to judge the intuitiveness of our system, the
suffering from neurological diseases affecting various participants were not given a significant amount of
functional components of the nervous system [5]. This guidance regarding how to utilize the system effectively.
study showed that users with damage to their nervous They also were not given any specific training tasks. We
system were able to utilize a BCI that recognized believe this led to most participants failing to gain a
sensorimotor signals at a comparable level to those complete understanding of the system. The Cognitiv input
without nervous system damage. Our goal is to provide an particularly was ineffective due to the training not
accessibility option to individuals unable to use a physical reflecting actual usage. The Cognitiv training focused on
keyboard. From our target group, we expect a significant using the eMotiv EPOC Control Panel rather than using
portion of users will have nervous system damage. the virtual keyboard. The significant difference is that the
While our error rate is higher than the studies participants were not required to utilize Cognitiv and
mentioned previously, our selection rate without using Expressiv inputs concurrently during training. We believe
Cognitiv input is lower than three of the techniques used a structured training process would improve both
in the related studies. However, our implementation must accuracy and responsiveness significantly. Furthermore, a
be improved to achieve a lower error rate and selection structured training process would provide participants
rate in order to realize our goal of providing a viable with a more thorough understanding of the system, which
accessibility option for users unable to use a physical should increase the participants’ comfort in using our
keyboard. We suggest a goal of 10 seconds per selection virtual keyboard.
and an error rate of no more than 15%.
5.2. Lessons Learned Regarding the System Design
5. Discussion of Lessons Learned
Several potential improvements to the design were
uncovered. The simplest of these was altering the display
In this section, we discuss lessons learned regarding both
to convey to the users that the modifier keys utilized a
the results of the user study as well as the design of the sticky key implementation (i.e., when a modifier key is
virtual keyboard. pressed, the modifier would remain active until the
modifier key was pressed again). This could be achieved
5.1. Lessons Learned Regarding the User Study by highlighting the modifier keys while they are active. A
more complex improvement uncovered was revealed by
We believe the limited amount of time allotted for training user frustration with the movement feature; specifically,
the participants was a significant hindrance to the the implementation where any movement key can be used
participants’ ability to accurately and quickly make to access any key on the keyboard if it is the only
selections. This belief is strongly supported by the movement key available. The current implementation
immediate feedback from users during the testing and on causes a movement key to produce a wrapping effect (i.e.,
 when the current key is at the end of a row or column, the 7. Conclusion and Future Work
movement key moves the current key to the beginning of
the next row or column). The wrapping effect was found We presented a novel system with the goal of producing a
to be disorienting for users when they bypassed the key viable alternative input device for those unable to use a
they wished to select. The users generally responded to physical keyboard. Our system is implemented using a
this situation by backing out of a section and restarting the BCI device; in particular, the eMotiv headset. We began
selection process. This response led most users to increase
by identifying a number of challenges that needed to be
the number of movements required to make the next
overcome for our system to be a viable keyboard
selection and increased user frustration.
alternative, and outlined how our design overcame these
Other potential improvements were suggested
challenges. We then presented a preliminary study that
directly by users. The suggested improvements include:
evaluated the current implementation of our system. The
 Adding support for macro keys to streamline the
study was followed by a user survey that gauged the
selection of common key sequences.
users’ satisfaction. The results of our preliminary study
 Changing the Shift key to not utilize the sticky
were very promising, but demonstrated a need for
key implementation due to the high number of
improvement in the current implementation. We then
situations that called for the Shift key to be
presented several studies of similar BCI systems and
activated and then immediately deactivated (e.g.,
compared our results to those found in the related studies.
when capitalizing letters).
We then discussed our lessons learned. Finally, we
 Some users suggested the addition of an input
discussed the threats to the validity of our study.
that deactivates and reactivates the system to
We believe our virtual keyboard, or a similar
prevent users from making unintended selections
implementation, can achieve our stated goal of an
during periods where the user is not actively
accessibility option for users incapable of using a physical
utilizing the system.
keyboard. Our current plans are to implement the
improvements outlined in the lessons learned section and
6. Threats to Validity to perform a more thorough study that incorporates the
lessons learned. We expect the enhancements to the
In this section, we present the identified threats to the system and an improved training procedure will increase
validity of our study. the selection rate and accuracy significantly.
6.1. Internal Validity
Acknowledgements
Users were required to first test the system with only the The authors would like to thank the participants in our
Expressiv suite active, followed immediately by testing user study. This work was largely supported by a
the system using both the Expressiv and Cognitiv suites. Department of Education GAANN grant (P200A100182).
This repeated testing could cause a fatigue effect that may
have contributed to the increased selection and error rates. References
We accepted this threat because each participant was only
[1] Emotiv | EEG System | Electroencephalography. ().
available for a brief time. Alternatively, the users could [2] H. Cecotti, Spelling with non-invasive Brain-Computer
have gained experience from the first test that would have Interfaces - Current and future trends. Journal of
carried over to the second test, therefore decreasing Physiology - Paris, 105, 1-3 (2011-01-00 2011), 106-114.
selection and error rates. With the limited number of [3] J. Yue, J. Jiang, Z. Zhou & D. Hu, SMR-Speller: A novel
subjects, the study asked all participants to use the same Brain-Computer Interface spell paradigm. In Anonymous
ordering of tasks. The group size would be too small to Computer Research and Development (ICCRD), 2011 3rd
International Conference on. (). , 2011, 187-190.
determine significance if the subjects had been separated
[4] D. J. McFarland & J. R. Wolpaw, Brain-Computer
into different orderings. Interfaces for Communication and Control Commun ACM,
54, 5 ( 2011), 60-66.
6.2. Construct Validity [5] C. Neuper, G. R. Muller-Putz, R. Scherer, & G.
Pfurtscheller, Motor imagery and EEG-based control of
The participants in our study were not from within the spelling devices and neuroprostheses Prog. Brain Res., 159(
target audience (i.e., our participants are able to use a 2006), 393-409. 6123(06)59025-9.
[6] E. W. Sellers, and E. Donchin, A P300-based brain-
traditional keyboard, but participated in our study by computer interface: initial tests by ALS patients Clin.
using only the virtual keyboard). We believe this is an Neurophysiol., 117, 3 (Mar 2006), 538-548.
acceptable threat. Comparing our results to those of [7] D. E. Thompson, J. J. Baker, W. A. Sarnacki, & J. E.
Neuper et al. [5] demonstrates our results are similar to Huggins, Plug-and-play brain-computer interface keyboard
those of the target audience. Furthermore, given the performance. In Anonymous Neural Engineering, 2009.
timeframe and emergent sensitivity of the device, it was NER '09. 4th International IEEE/EMBS Conference on. (). ,
2009, 433-435.
infeasible to involve users from the target demographic.

View publication stats