A team of researchers at Columbia University has developed a speech brain-computer interface system that translates brain signals into intelligible, recognizable speech. By monitoring someone’s brain activity, the system can reconstruct the words a person hears with unprecedented clarity. This could lead to new ways for computers to communicate directly with the brain, and lays the groundwork for helping people who cannot speak. Early efforts to decode brain signals researchers focused on simple computer models that analyzed spectrograms, which are visual representations of sound frequencies. But because this approach has failed to produce anything resembling intelligible speech, the team turned instead to a vocoder, a computer algorithm that can synthesize speech after being trained on recordings of people talking. This is the same technology used by Amazon Echo and Apple Siri to give verbal responses to our questions.
Researchers asked epilepsy patients already undergoing brain surgery to listen to sentences spoken by different people, while they measured patterns of brain activity. These neural patterns trained the vocoder. Next, they asked those same patients to listen to speakers reciting digits between 0 to 9, while recording brain signals that could then be run through the vocoder. The sound produced by the vocoder in response to those signals was analyzed and cleaned up by neural networks, a type of artificial intelligence that mimics the structure of neurons in the biological brain. The end result was a robotic-sounding voice reciting a sequence of numbers. To test the accuracy of the recording, the scientists tasked individuals to listen to the recording and report what they heard. They found that people could understand and repeat the sounds about 75% of the time, which is well above and beyond any previous attempts.