Eye Interfaces. Report in Yandex

In February, Yandex held the second conference, “I Frontend.” We made it unlike our traditional I. Subbotniks - almost all the speakers were not from Yandex so that viewers could learn about the experience of different companies and developers. During the day, 10 kg of marmalade was eaten, the guests watched the Yandex.Rover robot courier work, and in the evening they cheered for the participants of the Code in the Dark contest. As always, we will publish summaries of several reports from the conference. Ivan Bakaidov





ibakaidovworks as a programmer at LINKa in St. Petersburg. He is the creator of alternative communication programs and an advocate for the rights of people without signatures. Vanya spoke at a session of the Commission for the Deaf in Emergencies of the UN Humanitarian Summit in Istanbul. At our conference, he talked about “eye interfaces” - shared his experience in developing children's educational games and applications that are controlled by a gaze tracking device and designed for people with disabilities. From the abstract and video you will learn about the pitfalls in creating “eye interfaces” and you can answer for yourself the question of whether this is the future.


- Hi. They probably already introduced me, I don’t know, because I wrote this text yesterday, but today I just reproduce it with a speech synthesizer. Since I do not have this knowledge, I will introduce myself again. I’m Ivan Bakaidov. I am 21 years old. And my main occupation is that I make interfaces with which people with speech damage form phrases for a speech synthesizer. They can print text, select pictures, press one button, and all this should produce a full-fledged speech. My project is called LINKa .



But now I’m trying a new type of device for inputting information into a computer. This is an IT tracker, a device that tells the computer the point where a person is looking on the screen. It is small, mounted under the screen. And, it would seem, with a fairly good operation of the device, we can forget about mice and keyboards. But is it? This is my report.

My report has several goals:

1. To show you that the future is very close, eye control is already knocking on consumer doors and we should start sawing libraries for this area.
2. To tell about the main difference between such interfaces from the usual ones.
3. In general, paralysis at a conference is useful. Maybe one of you will look at me and go check the focus on the site so that I can use it as a tab.



In fact, for many years I was told about this technology, but for many years I did not believe in it. Firstly, the model of IT trackers cost about 140 thousand rubles, which is very expensive. Secondly, as you can see, my head is very twitching, and on the first models that I tried, it greatly interfered.



But now, Tobii has released a gaming tracker worth about ten thousand and with a head tracking function. That is, this device tracks the movements of the head and eyes and, I suspect, simply subtracts one from the other. It is worth mentioning that Tobii is like Apple, only in the market of trackers.

This tracker is designed for gamers and serves as an auxiliary function.



For example, if you look at the edge of the screen, the player’s camera turns slightly in this direction. Or in the latest shooters, if you look at opponents, they are marked on the map.



Another of the functions of this tracker, which comes out of the box, is help in the menu of open programs. That is, I hold on to the familiar alt + Tab, a window view appears, I look at the desired window and release alt + Tab. You get used to it instantly and then you don’t understand why this function does not work on your main macbook.

Oh yes, this thing turns on the computer when you sit down on the table and turns off when you leave for coffee.

By the way, the IT tracker works only with Windows, so everyone now looks at me as a sick person, because I’m running with a macbook on which Windows is. Although, if it weren’t for Windows, they would still look at me like a patient.



So, the program selection menu is cool, of course, but we want to control the system completely without a keyboard. We are people with motor disabilities. And there are two solutions. The first is obvious, you can control the mouse with your eyes. That is, the cursor moves to the gaze position, and when you hold the gaze, a click occurs.



The second solution is to create your own interfaces tailored for the tracker. As you know, if the first path worked normally, I would not talk about the second one.



There are several programs for the first path, I tried three:
- A tool built into the latest builds of Windows, which is simply included in the accessibility settings.
- Optikey program of the English developer.
- The program is from a Russian programmer, whose name the author clearly did not distinguish, so I will call it a trojan. Well, because the antivirus says it is a trojan.



Programs are generally similar in principle, so I will not disassemble each one. A virtual toolbar appears. You select one of these tools, for example, left-click, then look at the zone where you want to click, increase this zone, you look a second at a specific place and a click occurs. Raise your hand to those who understand that this is not our future.



Add to this the fact that my gaze twitches, because the eye is a muscle, and all the muscles of my body twitch. By the way, here I wrote down the trajectory of the gaze - mine and the ordinary person - while reading Lolita. As you can see, my view leaves the line up. Therefore, I have read seven books in my life and am afraid to go to college. I sent this picture to my literature teacher with the words: I am not a moron.



But in all these programs there is a function of the on-screen keyboard, which works on the principle of squares with letters. When you hold your gaze in a square, it is pressed. In this case, the box forgives the fact that you look away from him at the others for a couple of fractions of a second.

And you already understood what a bright thought occurred to me?

That's right, you need to make interfaces out of squares. Accordingly, I decided to write a component library for the IT tracker. Now I am preparing it for release. And according to the first demo applications, we can confidently say that the subject tracker, subject to the creation of special applications for it, will replace our mouse, since it is much more convenient to just look at the button.

I can’t say the same thing about the keyboard: after all, touch typing is more ergonomic and happens at a muscular level. In the case of the mouse, we always look at the screen where we click. And I think that very soon the interfaces will move in this direction.

In general, my vision is this. Five years later, IT tracking technology will tightly enter our lives, but these will be completely different interfaces. They will be a combination of a keyboard and tracker. That is, as in the example with alt + Tab, the user will open certain views using the keyboard shortcut, and select from them with a glance.

But I want to talk about my other project.

Since the skill of controlling the gaze is quite complicated for people, I started developing simple casual games that help me get the right skill. Actually, part of these games is tracing paper from its western counterpart for a lot of money. Now I will show the games and explain what skill they form in my opinion. The project is aimed at children with developmental disabilities.



The first game I made is called Flash. Its essence is very simple. Lenin appears on a black screen in a random place, and when looking into this area, Lenin changes to a light. Then the story repeats itself. Then I replaced the portrait of Vladimir Ilyich, they asked me.

With this game, the child begins to get acquainted with the tracker. Since there is only one figure on the screen, the child is likely to look at it. For this he receives a pleasant sound.

The game of the next level trains to keep a look on the object. The principle, again, is simple: you need to keep your eyes on the ball, it turns red, then bursts. The Vietnamese man gets a rocket on the head, and everyone is happy. The skill of holding the gaze is needed to control the buttons. And there is a whole series of games with buttons.

See the second game gif

But now I will show the game to the next skill level. This is a skill of oblique vision. It is needed at the time of typing, when you simultaneously need to look at the desired letter and follow the text that you enter. So I made a game of Sisyphus, where you need to gaze to set in motion a fan that blows on a ball. The ball must be rolled uphill. When you look away from the fan, the ball rolls back.

See the gif of the third game

Some such games have come out, this is my amateur project. Some of you may want to join him.

Contacts:
-ibakaidov
- ivan@aacidov.ru
- I want to study