In a key breakthrough for communication by people with impaired speech or motor skills, Stanford researchers have developed software that decodes neural activity from a device implanted in a person's brain while the person imagines writing letters and words by hand, realizing the fastest typing speed for any type of brain-computer interface at 90 characters per minute with over 94% accuracy in real time and over 99% accuracy when paired with autocorrect.

In a paper published Wednesday in Nature, the scientists proposed a brain-to-text artificial intelligence system that works in conjunction with a brain-computer interface, which establishes a connection between a wired brain and an external device. The study is one part of the larger, multi-institution clinical trial BrainGate2, which is evaluating the safety of the devices implanted in the brain that let people control computer cursors and other technologies by thought alone.

Francis R. Willett, a research scientist at Stanford University and first author of the paper, told The Academic Times that the intention of the study was to help people with severe paralysis who cannot easily speak or otherwise communicate. Their system was only tested with one participant, who has been paralyzed from the neck down since 2007, and who has participated in other BrainGate2 studies that involved tasks such as point-and-click typing and moving a robotic arm. 

The participant, referred to in the paper as T5, already had two brain-computer interface chips implanted in their brain as part of prior research. Each chip contains 100 electrodes that are able to pick up signals from neurons in the brain's motor cortex, which controls hand movement. Willett said he and his co-authors took a computational approach to develop a novel algorithm that could translate brain signals into text on a screen. 

In a series of sessions, T5 was asked to imagine that they were physically holding a pen and paper, and to mentally attempt to write out letters, words and, eventually, sentences. "As they do this, they will generate brain activity in motor areas of the brain that are trying to make this motion," Willett said. "And we're listening in on that activity."

The brain chips recorded the electrical activity of the neurons that were engaged while the participant imagined writing. Though the signals weren't reaching T5's hand muscles or making them move, they still generated brain activity. The researchers trained a recurrent neural network, which is a type of artificial intelligence network, to then take the brain activity information, look for patterns that represented what the participant was trying to write and translate those patterns into text on a screen. 

T5 repeated each letter 10 times to teach the software to recognize the neural signals associated with their attempt to write the letters. The study consisted of trials during which T5 copied sentences from an on-screen prompt and also responded naturally to open-ended questions. They were able to reach a writing speed of 90 characters per minute using the interface, which translated those characters into text with a 94.1% accuracy. This writing speed, unprecedented in brain-computer interfaces, is comparable to the average speed at which an able-bodied person in the same age range of T5 can type on a smartphone, 115 characters per minute.

"Previous [brain-computer interface] studies have shown that the motor intention for gross motor skills, such as reaching, grasping or moving a computer cursor, remains neurally encoded in the motor cortex after paralysis," the authors said in the paper, noting that it was "unknown whether the neural representation for a rapid and highly dexterous motor skill, such as handwriting, also remains intact."

"If you haven't moved your hand for 10 years, you haven't written anything for 10 years and your hand can't move well, when you try to do something like writing letters, does this even still evoke the correct patterns of neural activity in your brain?" Willett said.

The study's results suggest that it does, meaning that even years after a person is paralyzed, the neural representation of handwriting in their motor cortex could potentially be strong enough to enable communication via a brain-computer interface.

Currently, brain-computer interfaces are mostly used for restoring large movements in people with disabilities, such as reaching, grasping or controlling wheelchairs. Communication efforts with brain-computer interfaces have largely focused on relatively limited capabilities, such as point-and-click typing using two-dimensional computer cursors. Such studies have been successful, with participants reaching about 40 characters per minute with the point-and-click approach. Perhaps surprisingly, however, the results from this handwriting task reached more than twice that speed. 

"The reason we think this was able to actually go twice as fast is because when you handwrite these different letters, they evoke very different patterns of neural activity for each letter," said Willett.

The authors explained that handwritten letters may be easier to distinguish from each other than point-and-click movements because "letters have more variety in their spatiotemporal patterns of neural activity than do straight-line movements." Straight-line movements are used for clicking on a digital keyboard where letters are quite close to each other. 

Willett emphasized that further research is needed before the system can be deployed in real-world applications. "It's important to understand and emphasize that this is just a proof of concept demonstration, where we're demonstrating that this is an exciting approach that's potentially viable and we've demonstrated that it works," he added.

The software needs to be improved and streamlined for it to operate more efficiently. In the study, T5 and the researchers spent significant time teaching the software how to recognize the pattern of T5's mental handwriting, but for everyday use, it would need to run more quickly and with less training time. It also needs to be tested with more people, and should integrate a larger character set and enable text editing and deletion. 

Still, the authors said that their proposed system has considerable promise, as it is easy to use, able to express any sentence and accurate enough to someday help people communicate in the real world.

The study, "High-performance brain-to-text communication via handwriting" published on May 12 in Nature, was authored by Francis R. Willett, Donald T. Avansino, Jaimie M. Henderson and Krishna V. Shenoy, Stanford University; and Leigh R. Hochberg, Providence VA Medical Center, Brown University and Massachusetts General Hospital.