👩🏻‍🎤 🥈 👦🏽 Neurocomputer interfaces for games: what is known today 🎁 🧑🏿‍🤝‍🧑🏼 👨‍👩‍👦‍👦

Relatively recently, Valve co-founder Gabe Newell said that humanity is already much closer to the “Matrix” than it might seem. My co-author and I decided to find out if this is so, and why Valve, Facebook and Elon Musk are trying to connect the brain to the computer.

Illusion of control

A neurocomputer interface (NCI) or a brain-computer interface (BCI) is a system designed to exchange information directly between the brain and the computer. The concept was described by Joseph Liklider as far back as 1960 in an article with the sonorous title “ Man-Computer Symbiosis”. Now the publication is more likely to be of historical value, but active attempts to enter into such a symbiosis have been undertaken since the late 2000s.

Perhaps then NeuroSky showed itself more than others, together with Square Enix, which presented at the Tokyo Game Show 2008 a proof-of-concept game Judecca. In 2009, NeuroSky even launched commercial NKI board games, which you can still buy, but whether they read brain activity or exploit the phenomenon of the illusion of control is an open question.

The products of the rest of the companies basically did not go beyond the confines of the conferences, and the leaders of the gaming industry either kept silent, or said that they were not involved in the development of NKI.

NeuroSky's Mindflex and StarWars Force Trainer games are still the most successful BCI devices, despite the contradictions.

In 2012, the Laboratory of Neurocomputer Interfaces at the University of Graz in Austria introduced the NQI for World of Warcraft. The system worked, but obviously could not replace a mouse with a keyboard.

“Learning” the interface and the player: in order to associate a signal from the brain with a certain team, the user needs to first move his hand, and just think about this movement during the game.

Gradually, it became clear that a revolution would not happen. They forgot about NQI, but exactly until the famous Valve founder Gabe Newell said in an interview on March 18, 2020 that the company was conducting neurobiological research with might and main, adding that we are much closer to the Matrix than many think. Almost simultaneously, a report came outaccording to which the NKI market will be estimated at $ 2.96 billion by 2027.

What Gabe did not say

A year ago, a Valve employee, experimental psychologist Dr. Mike Ambinder, spoke at the Game Developers Conference 2019 about the development and applications of NQI. Compared to Gabe’s interview, almost no one noticed his speech, although he spoke much more about the company's plans.

Mike is in the OpenBCI EEG Headset , a community of enthusiasts developing open-source NKI solutions.

Valve has not yet taken the “Matrix” path, connecting directly to the brain, but using a special “helmet” for electroencephalography (EEG). This approach has drawbacks - namely, noise and low bandwidth (for example, the authors of the NCI for World of Warcraft estimated the data transfer rate at <100 bits / minute).

The main methods for obtaining information about brain activity, including using signal capture on peripheral nerves and muscles.

The disadvantages are partly due to the signal passing through the skin and bones. The EEG is usually compared to trying to understand what is happening on the football field, standing outside the walls of the stadium and listening to the cries of the fans.

Follow the match “from the stands” allow other methods:

Electrocorticography, ECOG - or intracranial EEG (intracranial electroencephalography - iEEG) - the most “soft” surgical method, when the electrodes are adjacent directly to the surface of the brain. Deep interference in its structure does not occur.
Intracortical recording (intracortical recordings) and implants - method with introduction of the electrode into the thickness of bark tend motor. Makes the electrodes even closer to the signal source, and the operation is even more complicated.

But the non-invasiveness of the EEG so far covers all the disadvantages, and Mike believes that its capabilities have not yet been exhausted. Already, it allows you to evaluate:

The player is keen on events on the screen or is about to die of boredom;
Tense during a difficult mission or relaxed and confident;
Is able to concentrate and absorb new information. Still, the gameplay involves training the player - management, the specifics of gameplay, etc. things;
Motor signals - with what movements and how intensely the player reacts to certain stimuli in the game.

And many other, more subtle parameters. Actively developing machine learning algorithms gradually make it possible to extract more and more from EEG data.

Do not rush to throw away the keyboard

Most likely, the way NKI in the gaming industry does not begin with immersion in virtual reality.

According to Mike, now NKI primarily allows you to get a new feedback from the player that can improve the quality of games. Mike believes that technology will go through three stages of development.

Each interface is a brain-computer interface. The only difference is the intermediate hardware link with which we translate our intentions into something suitable for use. Dr. Mike Ambinder, experimental psychologist Valve.

In the short term, NKIs will find application in game development and make the testing process more efficient. Now the usual method for determining the emotions of a tester is a survey. At the same time, it can be carried out only after passing the game level, when some impressions are already forgotten, smoothed or rethought. Reading real-time brain responses to an in-game stimulus is much more objective and will help to better design levels, or rather fall into planned emotions.

An analysis of eye movement, heartbeat, muscle tension and even posture is a way to assess a player’s condition without touching the brain. This may be of no less interest than the EEG.

Gradually, NCI will become more accessible and go beyond the studios. At the second stage, the game itself will get access to the data. This, according to Mike, will completely change the concept of adaptive gameplay. Traditional difficulty levels will become obsolete - the game world will adapt to a specific player, determining which enemies cause him more problems, whether he wants to relax after a hard day or is bored in anticipation of a difficult battle.

Are users ready to share their status and even thoughts with the game or find it too personal?

Ultimately, the characters in the game will be able to respond to the player’s mood and emotional state, making decisions based on these parameters.

Mimicry in LA Noir (2011) allows a player to guess if a character is lying or tells the truth. With the advent of NKI, roles can change.

, , , . RPG, . (Mike Ambinder), Valve.

“True NCI” - replacing standard, “physical” management interfaces will be the last step. This will solve the problems of modern interfaces: a long reaction time, the need to remember keys, the inability to press many keys at once. But he will add new ones: for example, the lack of tactile feedback. Having missed the “jump” key, it is not difficult to understand why the character did not jump. With the mentally given command, the same situation is no longer so clear.

The final stage in the development of NQI is the creation of the very “Matrix”: data output devices directly to the brain, bypassing monitors, VR helmets and audio systems. According to Gabe, theoretical studies are already underway and bring a lot of surprises: externally “complex” tasks - “connecting” to the motor cortex - turn out to be relatively simple, while others - for example, imitating a feeling of cold - are incredibly complex.

It turns out that the brain has very good built-in interfaces for some things, and very bad ones for others. Gabe Newell, co-founder and CEO of Valve

TMS device used by Valve.

At the same time, climbing inside the skull is still not necessary. For primary research, transcranial magnetic stimulation is sufficient - a method in which certain brain structures are affected by magnetic impulses, causing deep stimulation. Now this approach is used mainly for the treatment of certain mental and neurological diseases, for which TMS is specifically used in Valve Mike did not specify.

Brain implants. How do you like it, Gabe?

Much of what Mike described can be implemented using EEG. But he admits that someday implants will become as commonplace as LASIK - an operation for laser vision correction. Mike calls this point “LASIK-threshold”. Elon Musk is going to overcome it, whose company Neuralink has already introduced the N1 implant, promising to begin clinical trials in 2020.

A robot surgeon created at Neuralink , capable of implanting up to 192 electrodes per minute with an accuracy of several microns.

The phased implantation of filaments with electrodes by a robot into a substrate simulating brain tissue. Neuralink did a great job creating 20 types of threads for different tasks.

According to Philip Sabes, a Neuralink employee at the University of California at San Francisco, the project’s main goal is to help paralyzed people. Including - returning the opportunity to play sports, at least virtually, through games.

Some invasive NQIs have already been tested in humans. The Braingate intracortical implant in 2008 allowed a completely paralyzed patient to control the cursor, and already in 2012 - to take a bottle with a robotic arm from the table and take a sip. With Neuralink technology, the capabilities of these interfaces can be expanded.

Neuro interfaces can help people regain hearing, vision, improve prostheses and even restore the functions of paralyzed parts of the body.

Primary data processing ASIC Neuralink module right inside the cranium.

Data output from the brain in the case of an invasive implant is no less complicated than with EEG. In order for the information not to be distorted, the primary processing, signal amplification and noise suppression are best done as close as possible to neurons. For this, Neuralink has developed special plug-and-play modules that can be placed directly inside the patient’s cranial box. They connect to standard interfaces like USB-c and, in the future, Bluetooth, but their main advantage is scalability, which allows you to connect a large number of modules at once.

Ilon himself, as always, formulates goals even more boldly: he wants to create a “full-brain NQI”, providing the very symbiosis of a person with artificial intelligence. At the same time, the businessman admits that even with all the achievements of Neuralink, this is still fantastic.

Closer to reality

In November 2019, Facebook acquired CTRL-labs. Perhaps CTRL-labs attracted attention by the fact that they plan to reach the main goal of any NQI - the brain - not at all through the holes in the skull. And this approach looks more realistic than the others.

One of the earliest versions of the CTRL-kit.

CTRL-labs director Thomas Reardonexplains that the Neuralink implant is good for paralyzed patients, but a healthy person already has a ready-made neural interface - nerve endings and muscles connected to them. Non-invasive methods are ideally suited here: the activity generated by the muscles is several times stronger than the signal from any brain structure. The most complicated EEG decryption algorithms are practically not required - the main work has already been done by the human body. Therefore, the CTRL-kit introduced in 2018 is not a “helmet”, but a bracelet for electromyography .

One of the demo "games" CTRL-labs. The company's developments are so far more like a game-ready interface than anything that works on the basis of EEG.

The interface, similar to the CTRL-kit, appeared in 2009. But the similarity is only external: such a controller does not work without muscle contraction.

At the same time, the CTRL-kit can be considered a complete brain-computer interface that reads the activity of neurons. To prove this, a dinosaur from Google Chrome was connected to the bracelet.

Demonstration of using the CTRL-kit to control a dinosaur from Google Chrome.

According to Philip, it takes an average of 90 seconds to study. At the first stage of training, the player simply presses a button. After - it imitates pressing with the help of muscles - first with difficulty, then - almost reflexively. And here the most interesting thing happens: the muscles are excluded from the chain.

: “ , ?”. . (Thomas Reardon), CTRL-labs CEO.

At some point, the answer is: there is no need to contract muscles, just think about movement - this impulse is enough for the CTRL-kit to react - the dinosaur will jump.

Take a cup from the table and bring it to your mouth - what could be easier? But the task is simple only at first glance - each time the distance to the cup, its material, weight, shape and other parameters change. An adult easily adapts to these changes - as well as to different options for using the CTRL-kit. A study of how motor adaptation occurs is useful in robotics - machines do not yet know how to adapt so well to changing conditions.

The interface partly uses the principle of " motor babble"- chaotic movements with the help of which the baby makes a" map "of the body, linking the activity of certain areas of the brain with movements. Gradually, the process of such training improves, but never stops. Ctrl-labs essentially redirect this ability to adapt to the development of motor interfaces.

The CTRL-kit doesn’t just work - it helps to recreate the chains in which neurons are combined. The main breakthrough, according to Thomas, is that CTRL-labs technology can detect the activity of individual neurons. Evidence of this has not yet been published, but the company promises to release an article in the near future, and if it turns out to be convincing, then the main goal of neurobiology - deciphering how the brain works, will become much closer, as well as creating NQI without mediation of muscles and peripheral neurons.

It is worth noting that in Valve, apparently, they are also interested in the peripheral nervous system. Mike mentions this approach as a potential way to reduce response time. For example, if the signal from the brain to the fingers reaches 100 milliseconds (the number is taken as an example), then the “interception" of the signal on the approach to the muscle will help save 10-30 milliseconds. At first glance, this is insignificant, but such “cheating” will surely please e-sportsmen. True, while interfaces can only increase response time, but technology does not stand still.

What will happen next

Science fiction has posed many ethical issues related to NQIs, but the post-NQI world will certainly pose even more.

Gabe is confident that technology will find application in all sectors of the entertainment industry, but is skeptical of forecasts. According to him, presenting the post-NQI world now is like trying to explain what the Internet is to a person who has never heard of it. “Cool, you can look for recipes there!” - Such a person will say, but he will not be able to predict the appearance, for example, of Uber.

Voluntarily forcing gamers to buy new devices is not so simple. The expensive development of NQIs in this regard is a huge risk. Screenshot from Half-Life: Alyx (2020).

Nevertheless, all developers say that NII will become a logical development of virtual reality technologies. Making a player wear a helmet is not an easy task, but adding NKI electrodes to the VR headset can dramatically change the feel of the game and make the device attractive.

There have already been attempts to create hybrid helmets - Neurable has shown such a device at several exhibitions and even announced the game Awakening, but the release has not yet happened.

Virtual reality is mentioned in the speeches of employees of all companies. But what no one said was how much to wait. Neither Mike Embinder nor Gabe Newell announced the release date for Half-Life 3 on NKI. Philip Sabes and Elon Musk, although they mention the game, but the invasive implant is unlikely to immediately replace the gamepad. Only Thomas Reardon names specific dates - and given the realistic approach of CTRL-labs, there is no reason not to believe him. According to him, the first mass NKIs will not appear on the market until 2024.

Additional materials:

For those who are eager to try out the NQI, we have compiled a list of links with theory, as well as with devices and applications that are already being sold or being prepared for release.

History and future of BCI:

openbci.com — -, open-source . , .
store.neurosky.com – Neurosky, MindFlex StarWars Force Trainer, , , , .
emotiv.com – Neurosky, , -. « » . .
brainco.tech – BrainCo, . , .
next-mind.com - NextMind startup, interesting primarily because their upcoming release devkit is the most compact and mobile EEG device. Also, developers promise compatibility with games.

Authors:

Ivan Smirnov (I), concept artist, illustrator, founder of Smirnov School.
Artemy Tretyakov, geneticist, author at Smirnov School.

Neurocomputer interfaces for games: what is known today