🔪 🚶🏽 🕐 How to reproduce realistic sound in computer games and VR and why it is difficult 🤼 👵 👨🏼‍✈️

The calculation of the acoustic parameters of an object in a virtual environment can take days. At Stanford, they developed a system that reduces this time to several minutes and even seconds.

We tell you how the algorithm works, allowing you to create more realistic worlds in games.

^{Photo Stella Jacob / Unsplash}

Sounds in a virtual environment - it's hard

A significant part of the technologies associated with the generation of sound models is based on the work of the German physicist German von Helmholtz (Hermann von Helmholtz). As early as the 19th century, he proposed the theory of acoustic resonance and derived an equation describing the propagation of sound in space. Today, his scientific research forms the basis of algorithms that simulate the movement of waves in virtual environments, taking into account volume and direction.

One of the well-known methods is the boundary element method (BEM). It helps to solve the problem of sound scattering by solids of complex shape, but requires preliminary calculations of the parameters of objects. The procedure can take several hours or days, depending on the computing power of the cluster. This undoubtedly affects the cost of the product, whether it be an animated film or a computer game.

Engineers from Stanford University took up the problem. They presented an algorithm that can build an acoustic model of an object in a virtual environment in a few minutes - without using the Helmholtz equation and BEM. The technology is called KleinPAT, in honor of the Austrian composer Fritz Heinrich Klein ( Fritz Heinrich Klein), which in 1921 brought out the “ mother chord ” - a chord of twelve different sounds separated by eleven different intervals.

How the algorithm works

The project is based on the method ( page 1 ) of modal sound synthesis, which reduces the time spent on preliminary calculation of the acoustic parameters of a dynamic model and supports hardware acceleration on the GPU. The solution of Stanford engineers does not calculate the frequency characteristics of a 3D object for all types of vibrations. Instead, like Heinrich Klein, he combines several models into so-called chords .

From these chords, the algorithm extracts the frequency transfer functions for each type of oscillation using a special QR decomposition system . On their basis, the necessary acoustic fields are generated - thousands of times faster than in the case of methods based on the Helmholtz equation. As it sounds in practice, you canlisten to the video that the developers recorded.

^{Photo by Stéphane Bernard / Unsplash The technology}

is expected to find use in VR projects. There are a lot of variables in games of this genre, the environment is constantly changing. In such conditions, it is difficult to generate realistic sound. But according to the developers, their algorithm will allow you to calculate the acoustic parameters of the virtual environment in almost "live broadcast". There is also an opinion that the new solution is suitable not only for working with objects, but also for human speech.

So far, KleinPAT is locatedin the early stages of development, so it will take some time before the algorithm reaches commercial VR-projects. At the same time, one of the residents of the TechSpot site noted that support for GPU computing played a significant role in improving productivity. He suggested that classic BEM methods, optimized for working with graphic cards, could also show decent results. Perhaps in the future, research will also be conducted in this direction.

Who else works in this area

The development of algorithms related to the simulation of sounds in virtual space, and involved in large IT companies. For example, there is the Triton project , which simulates the movement of waves in complex 3D spaces: with a large number of angles, rooms, objects. The system “ bakes ” (as in the case with texturing) acoustic parameters of the environment on static geometry, and then reproduces them in the game. Designers retain the ability to fine-tune the sound - for example, reduce reverb so that the characters' speech is more legible. All this processing takes only 10% of the processing power of the CPU. The Triton technology has already been used in the computer game Gears of War - it adds a noticeable echo when communicating with the main characters in the big hall.

^{PhotoBartek Mazurek / Unsplash}

Acoustic technology alsoworksat Intel. With the support of the company, composer Justin Lassen travels the world and makes binaural audio recordings. On them, he seeks to capture the sounds of the environment. Records made in this way turn intoassetsfor the Unreal Engine 4 engine. Later they are used in the development of VR projects.

In the future, such solutions will create virtual worlds that will immerse the player with what is happening on the screen.

— « Hi-Fi»:

«-, »:

—

« »:

How to reproduce realistic sound in computer games and VR and why it is difficult

Sounds in a virtual environment - it's hard

How the algorithm works

Who else works in this area

More articles: