Picture: 123RF/andreykuzmin
Picture: 123RF/andreykuzmin

Researchers at Columbia University in New York have published the remarkable results of ongoing research into "mind-reading" technology that can synthesise speech from brain activity.

The paper is titled "Towards reconstructing intelligible speech from the human auditory cortex", and was published by Scientific Reports on January 29.

The full text is accessible from Nature.com.

The neuro-engineers behind the breakthrough work at the university’s Zuckerman Institute — named for media mogul and investor Mortimer Zuckerman — which focuses on the brain and mind.

Nima Mesgarani, the principal investigator and lead author of the paper, explains how researchers used a combination of a vocoder and electrocorticography on five volunteers for this project.

The vocoder here is a computer algorithm that learns to "speak" from a data stream and through exposure to language, as seen in virtual assistants like Siri and Amazon’s Echo, while with electrocorticography patients have electrodes implanted in their brains to measure brain activity.

The volunteers were undergoing brain surgery for epilepsy, and consented to having the implants placed during this process.

Initially they were exposed to a short selection of spoken sentences to identify patterns in their brain activity while listening. This data was used to train the vocoder.

Thereafter the patients heard prerecorded recited numbers between zero and nine.

The resulting brain activity data was then run through the vocoder to produce sound. This, in turn, was analysed and cleaned up, and the final output is a robotic- sounding voice reciting the numbers — stilted and early- sci-fi-esque, but clearly audible.

The recordings are uploaded as "supplementary materials" at the end of the paper on Nature.com, so if you want a spine-tingling experience of a computer voice powered by thought, this is where you’ll find it.

Next, the Zuckerman team plan to move on to "more complicated words and sentences", with a view to working towards an implant "that translates the wearer’s thoughts directly into words".

In the release, Mesgarani calls this a "game-changer" which "would give anyone who has lost their ability to speak, whether through injury or disease, the renewed chance to connect to the world around them".

This is just one of many research projects aimed at connecting the human brain to technology. Mesgarani is also involved in developing a hearing aid that "reads" brain activity to determine which voice a subject is most intent on hearing and "focusing" on that sound.

At the University of Toronto Scarborough, Canada, neuroscience researchers have used electroencephalography and technology to re-create images shown to people, including a reasonably accurate re-creation of faces.

Another exciting area of research is mind-controlled prosthetics, such as that demonstrated by engineers at Johns Hopkins University in 2016, where a volunteer was able to control the movements of individual fingers on a prosthetic arm through a series of sensors placed on the brain.

Not to be left behind, there is also a handful of neuroscience start-ups launching, such as Kernel, which hopes to use implants to address neurodegenerative disease.

Tesla founder Elon Musk entered the fray a few years ago, founding Neuralink. Here, scientists are reportedly working on developing brain-computer interfaces aimed at merging machines and brains.

For now though, we don’t know much about the work at Neuralink. The home page of its website (neuralink.com) is just a list of the kinds of people and skills it needs, and an address to send your CV to.

The medical and knowledge implications of these types of advances are exciting and terrifying in equal measure.

They offer a future of "hacked humanity", where we can enhance our senses, shrug off devastating injuries, and use our brains to control the physical world around us.

But, at the risk of sounding a tiny bit paranoid, there is something to be said for the fears of conspiracy theorists here.

For example, scientists from Kamitani Lab at Kyoto University, Japan, have used technology to draw images of recently seen scenes based on brain activity.

This, as well as the Zuckerman and Canadian examples above, could arguably be used to extract information against a person’s will, and there have been demonstrations of how connecting two people to a system can allow one person to control the basic motor functions of the other using input and output electrodes.

Though we’ve had the ability to detect and visualise brain activity (through metrics like electrical impulses and blood flow) for a while, this, combined with advances in analytical tools such as artificial intelligence, machine learning and neural networks (computing systems with nodal structures inspired by the brain and nervous system) has meant great leaps in the functionality of these technologies.

The widespread use of this kind of power is a way off still, but for now you can console (or terrify) yourself with the fact that mind-controlled virtual reality games, drones and cars have already been successfully demonstrated.

As author William Gibson says: "The future is already here, it’s just unevenly distributed."