Get best of the week

Why AI requirements can only make matters worse

By creating more transparent neural networks, we can begin to overly trust them. It may be worth changing the methods by which they explain their work.




Apol Esan once on trial took a ride in a robotic car from Uber. Instead of worrying about the empty driver's seat, passengers were asked to watch a “calming” screen, which showed how the car saw the road: dangers were drawn in orange and red, safe areas in dark blue.

For Esan, who studies the interaction of people with AI at the Georgia Institute of Technology in Atlanta, the message that they tried to convey to him was understandable: “Don’t worry, these are the reasons why the machine behaves this way.” However, something in the alien image of the street did not reassure, but rather emphasized the strangeness of what was happening. Esan wondered: what if the robomobile could really explain?

The success of deep learning is based on uncertain picking in the code: the best neural networks adjust and adapt to improve them further, and practical results overtake their theoretical understanding. In summary, the details of how the trained model works are usually unknown. We are already used to thinking of AI as a black box.

And most of the time it suits us - when it comes to tasks such as playing go, translating text, or picking up the next series with Netflix. But if AI is used to help make decisions in areas such as law enforcement, medical diagnostics, and robotic vehicles, then we need to understand how it makes its decisions and know when they turn out to be wrong.

People need the opportunity to disagree with an automatic solution or reject it, says Iris Hawley , an IT specialist at Williams College in Williamsstown, Massachusetts. And without it, people will resist this technology. “Already now you can observe how this happens, in the form of people's reactions to facial recognition systems,” she says.

Esan is part of a small but growing group of researchers trying to improve AI's ability to explain and help us peer into the black box. The purpose of creating the so-called interpreted or explained by AI (III) is to help people understand on what signs of data the neural network is really learning - and decide whether the resulting model turned out to be accurate and unbiased.

One solution is to create machine learning (MO) systems that demonstrate the insides of their work - the so-called aquarium AI, instead of the AI ​​in the black box. Aquarium models are usually radically simplified versions of the NS, in which it is easier to track how individual pieces of data affect the model.

“There are people in this community who urge the use of aquarium models in any high stakes situation,” saysJennifer Worthman Vaughn , IT Specialist at Microsoft Research. “And overall, I agree.” Simple aquarium models can work just as well as more complex NSs, on certain types of structured data, such as tables or statistics. And in some cases this is enough.

However, it all depends on the area of ​​work. If we want to learn from such fuzzy data as images or text, we have no choice but to use deep - and therefore opaque - neural networks. The ability of such NSs to find a meaningful connection between a huge number of disparate features is associated with their complexity.

And even here, aquarium MO can help. One solution is to go through the data twice, training the imperfect aquarium model as a debugging step to catch potential errors that you would like to fix. After cleaning the data, you can also train a more accurate model of AI in a black box.

However, such a balance is difficult to maintain. Too much transparency can cause information overload. In a study from 2018, which examined the interaction of untrained users with MO tools, Vaughn found that transparent models can actually complicate the search and correction of model errors.

Another approach is to include a visualization that shows several key properties of the model and the underlying data. The idea is to identify serious problems by eye. For example, a model may rely too heavily on certain attributes, which may be a signal for bias.

These visualization tools have become extremely popular in a short time. But is there any use for them? In the first study of this kind, Vaughn and the team tried to answer this question, and eventually found several serious problems.

The team took two popular interpretive tools that provide an overview of the model with the help of graphs and charts, where it is noted what data the model mainly paid attention to during training. Eleven AI professionals with various backgrounds, backgrounds, and backgrounds have been hired from Microsoft. They took part in a simulation of interaction with the MO model, trained on national income data from the 1994 United States Census. The experiment was specifically designed to simulate how data scientists use interpreting tools to perform their daily tasks.

The team found something amazing. Yes, sometimes tools helped people find missing values ​​in the data. However, all this usefulness has faded in comparison with the tendency to excessive trust in visualizations, as well as errors in their understanding. Sometimes users could not even describe what exactly the visualizations demonstrate. This led to incorrect assumptions regarding the data set, models, and the interpreting tools themselves. It also inspired false confidence in the tools and aroused enthusiasm for putting these models into practice, although sometimes it seemed to the participants that something was going wrong. Which is unpleasant, it worked even when the output was specially tweaked so that the explanations of the work made no sense.

To confirm the findings, the researchers conducted an online survey among 200 professionals in the field of Moscow, attracted through mailing lists and social networks. They found similar confusion and unfounded confidence.

To make matters worse, many survey participants were willing to use visualizations to make decisions about model implementation, despite recognizing that they did not understand the underlying mathematics. “It was especially surprising to see people justify the oddities in the data by coming up with explanations for this,” says Harmanpreet Kaur of the University of Michigan, co-author of the study. “The distortion of automation is a very important factor that we have not considered.”

Oh, this is a distortion of automation. In other words, people tend to trust computers. And this is not a new phenomenon. From airplane autopilots to spellchecking systems, everywhere, according to research, people often tend to trust system solutions, even when they are obviously wrong. But when this happens with tools specifically designed to correct just this phenomenon, we have an even bigger problem.

What can be done about this? Some believe that part of the problems of the first wave of III is connected with the fact that researchers of the Ministry of Defense dominate in it, most of which are experts using AI systems. Tim Miller from the University of Melbourne, studying the use of AI systems by people: “This is a mental hospital under the control of psychos.”

This is what Esan realized in the back seat of a Uber car without a driver. It will be easier to understand what the automated system does - and see where it is wrong - if it explains its actions in the way a person would. Esan and his colleague, Mark Riddle , are developing an MO system that automatically generates similar explanations in natural language. In an early prototype, the couple took a neural network, trained to play a classic game from the 1980s, Frogger, and trained it to give explanations before each move.


among the cars ... I can’t get through ... I'll wait for the gap ...

To do this, they showed the system many examples of how people play this game, commenting on actions out loud. Then they took a neural network that translated from one language to another, and adapted it to translate game actions into explanations in a natural language. And now, when the National Assembly sees an action in the game, it “translates” it into an explanation. The result is an AI playing Frogger that says things like “move left to be behind the blue truck with every move.”

The work of Esan and Riddle is only the beginning. Firstly, it is not clear whether the MO system will always be able to explain its actions in natural language. Take AlphaZero from DeepMind playing the go board game. One of the most amazing features of this program is that it can make a winning move that human players could not even think about at that particular moment in the game. If AlphaZero could explain its moves, would that be meaningful?

Reasons can help, whether we understand them or not, Esan says: “The goal of an III with a focus on people is not just to make the user accept what the AI ​​says - but also cause some thought.” Riddle recalls watching a broadcast of the match between DeepMind AI and Korean champion Lee Sedol. Commentators discussed what AlphaZero sees and thinks. “But AlphaZero doesn't work that way,” says Riddle. “However, it seemed to me that the comments were necessary to understand what was happening.”

And although this new wave of III researchers, I agree that if more people use AI systems, then these people should take part in the design from the very beginning - and different people need different explanations. This is confirmed by a new study by Howley and her colleagues, in which they showed that people's ability to understand interactive or static visualization depends on their level of education. Imagine AI diagnosing cancer, Esan says. I would like the explanation he gives to the oncologist to be different from the explanation for the patient.

Ultimately, we want AI to be able to explain not only to scientists working with data and doctors, but also to policemen using an image recognition system, teachers using analytic programs at school, students trying to understand the work of tapes on social networks - and before any person in the back seat of a robomobile. “We always knew that people tend to overly trust technology, and this is especially true for AI systems,” says Riddle. “The more often you call the system smart, the more people are confident that it is smarter than people.”

Explanations that everyone could understand could destroy this illusion.

More articles:


All Articles