A man who lost the ability to speak can now hold real-time conversations and even sing through a brain-controlled synthetic voice. The brain-computer interface reads the man’s neural activity via electrodes implanted in his brain and then instantaneously generates speech sounds that reflect his intended pitch, intonation and emphasis. To synthesise speech more realistically, researchers implanted 256 electrodes into the parts of the man’s brain that help control the facial muscles used for speaking. Then, across multiple sessions, the researchers showed him thousands of sentences on a screen and asked him to try saying them aloud, sometimes with specific intonations, while recording his brain activity.
Next, the team fed that data into an artificial intelligence model that was trained to associate specific patterns of neural activity with the words and inflections the man was trying to express. The machine then generated speech based on the brain signals, producing a voice that reflected both what he intended to say and how he wanted to say it. The researchers even trained the AI on voice recordings from before the man’s condition progressed, using voice-cloning technology to make the synthetic voice sound like his own. In another part of the experiment, the researchers had him try to sing simple melodies using different pitches. Their model decoded his intended pitch in real time and then adjusted the singing voice it produced.
More information: