Get best of the week

How our unmanned tram sees the real city

Hello, Habr!

In general, there is an experimental tram, which sometimes went along one of the routes as part of the tests. The autopilot is tested in a closed area, and in urban areas - an active assistant to the driver of a car driver. The tram driver rides with his hands on the control, but it is the autonomous autopilot that is being tested. The tram is visually no different from the usual one, because together with the manufacturer we crammed the instrument blocks far below the panels and brought the interfaces to standard screens. The only thing - he can see several cameras under the windshield, hidden under the skin of the radar and a GPS sensor on the roof. Yes, sometimes we hang lidar for debugging purposes.

image

During the tests, we learned that the rules of the road and the real situation on the roads, even for a tram, are very different things.

In general, a tram is an ideal sandbox for a complete autopilot of a car. We have already implemented it. Our cheats:

  • We know the route and have a guarantee that our vehicle will not go anywhere with it.
  • You can drive ahead and mark the points with traffic lights and so on, so that it is easier for the system to recognize them.
  • A tram cannot change lanes to lanes. Most of the autopilot’s load is tied to “where to taxi now” and thousands of scenarios, and we have nowhere to taxi.
  • It slows down almost instantly and a little sharply, that is, forecasts of the movement of other vehicles on the road are less complicated.

What really is the problem is with people at stops who try to climb first, risking their lives.

Test stages


In the first two stages we did autopilot in closed areas (plant \ depot \ site in front of the depot). A tram can move along a route, stop at stops and open doors, let pedestrians, cars pass and stop in front of obstacles. At the same time, we developed an assistant driver who was already tested in urban conditions. The assistant consists of two main subsystems. Speed ​​limit (OS): ensures that the driver does not exceed the speed (for example, according to the rules for technical operation of trams, bridges cannot be driven faster than 20 km / h, and in tunnels - 10 km / h. Arrows must be passed at a speed of 5 km / h, etc.). The other is auto-braking (AT), making sure that the driver does not crush anyone: it slows down (until it stops completely) in front of road hazards, red traffic lights,closed arrows. In this case, the control of the tram occurs in conjunction with the driver. This allowed us to smoothly test different parts of the drone in successively more difficult conditions.

Finally, we are now in the process of testing the autopilot on a live route. As part of the assistance system tests, we ride the real route at the moment, but our tram does not pick up passengers at stops and often does not stop at them. This is not visible from afar, and no one reads the inscription “Running in”, especially in places where there is a single 17th route. Therefore, often the people, assuming that we will be standing at the bus stop, boldly go against us.

Those same people


If in the case of co-pilots of agricultural machinery and combines it is important for us to ensure the maximum efficiency of grain harvesting and not to fly into a tractor, then the situation is different. It's about efficiency, but the main thing is not to bring down anything. And nobody. This also has an economic justification, because a tram that got into an accident is not only a cost for its repair, but also a simple expensive vehicle in a depot, that is, lost profit, and also often a simple whole line for a long time.

On the other hand, we have a radar and already three cameras to keep an eye on what is happening. This means that we can very accurately identify obstacles.

The problem is that the tram often intersects with people. In Moscow, people stand right on the edge of the path on the platform and seem to sway in the wind.

Our collision avoidance model does the following conditionally:

  • Detects obstacles and calculates their trajectories, possible behavior.
  • It determines the moments of intersection with our trajectory and the time before that.
  • It reduces speed in advance and notifies or slows down urgently.
  • All this is complemented by a radar search of obstacles ahead.

image

Pedestrians in the model are things that can move at speeds of up to eight kilometers per hour on foot, jogging or jumping. They have almost no inertia, they have small dimensions - in general, a solid random. And in no case should they be crushed.

Specifically, we are currently working on a more promising multi-hypothesis model. We understand that pedestrians in this place can either walk along the platform, or stand on it, or cross the road along this path. Well, if none of the models fits, then already in this case a person is classified as inadequate, and then yes, a solid random, we shy away from him in all directions.

The first firmware releases could not drive to a halt: unpredictable people were standing too close. The tram driver turned off the co-pilot on that trip at each stop.

You can, of course, reduce the possible level of anxiety of the model near the stop (we know where it is, the route is fixed) and increase it along the way. This is a good option in order to solve the porch problem, but ineffective if some hero nevertheless jumps out of the way.

I had to build skeletal models of each individual pedestrian (including in the crowd, which is not a very trivial task) and monitor the position of the joints. Now the tram thinks that most pedestrians rarely travel with their backs forward. We collected a more or less interesting sample of pedestrians running across the road and trained the neural network to see the intention in characteristic positions. It turned out well. Glory to modern technologies that allow cramming as much computing power into a small box as on the entire planet at the dawn of the space age.

We have stops - at ground level (taking into account the terrain along the route), and we already know how to predict movement well. Not knowing this feature, we asked for the preparation of the first prototype in 2018 to make an emergency, counting on an ordinary hard stop. I hit and then flew into the salon. The driver said: "I warned you: hold on tight." Passengers find this very rarely; I am sure that with emergency braking, almost the entire interior will fall.

Traffic lights


When a driver arrives on a new route, they “drive him” into it: he makes one or several trips along the new route with an experienced carriage driver, who rolled this route hundreds of times. We also “roll” the system into the route, mark it in advance, which ensures high accuracy of the system. The second unpleasant feature of the route - drivers make the same mistakes that the autopilot in the first tests without marking the route, trying to recognize the signs and control their movement. This is encouraging. By marking the route, accuracy was achieved.

Knowing the exact position of the traffic lights allows us, firstly, to lower the requirements for the accuracy of the detector (if it is mistaken where 100% traffic is not there, it is not very critical), and secondly, in deciding on the reaction to the position of the traffic light. At a particular intersection, we know for sure which of the traffic lights belongs to us (and which does not apply), where it has a stop line, how much it has a green period, and what kind of signal is permissive for us.

If we don’t know the topology of the intersection, then, having passed on the flashing green, we can start to slow down on the red traffic light-understudy located on the far side of the intersection. But in our system this is taken into account: the distant traffic light regulates the passage precisely through the beginning of the intersection, and not through the end.

If this is a prohibition signal, then you need to calculate the time of arrival at the traffic light and the possible speed.

Ambient conditions are very important for emergency braking. If the rails are wet or icy, if it is not correct to brake properly, the wagon will go by, you will either have to cut the rail brake (by the tram, by the way, there are three different brakes), or try to smoothly take the tram out of the skid and brake in the normal way. A rail brake breaks paths, for example. Actually, therefore, our speed profile (or calculation of the braking distance) takes into account these conditions, otherwise you can fly out of the way \ go to the user.

image

Calculation of the stopping distance is done from the curvature of the track, a specific section of the route, the state of coverage. This means that it is harder to slow down in the rain, and easier in the heat. The tram “knows” the weather by recognizing what is happening on the street and reacting to the first cautious actions at the beginning of the route and along its course (that is, the co-pilot will react to the onset of rain). There are several traffic profiles (we collected them from fixed-line trams, so we know who usually exceeds, but Mosgortrans, of course, will not “hand over” drivers: they helped us a lot) - from “rain” and “snow” to “sun” and "Ice". You can pre-select and firmly pin any of them from the interface. By default, the tram is aware of the environment itself.

So, if the driver does not respond to the prohibitory traffic light, the tram will first start to slow down, and then gently brake.

Here we were waiting for the second block of surprises. The co-pilot's early releases were careful and stopped the tram a few meters from the traffic light. Drivers sadly shook their heads and said that in Russia such an autopilot is doomed. It turned out that you need to get up more tightly, otherwise, for example, the tail of a long tram can stick out at the intersection. As a result, now the robot asks for several runs of the route with the operator in order to learn all the features, and knows which traffic lights you need to get up tightly.

Crossroads


The next portion of woeful shaking of the head befell us at the crossroads during rush hours.

The tram, which in the morning passed the intersection with a bang, was stupid.

It turned out that at a busy intersection in the heads of drivers - neural networks, in which the priority "drive faster" will prevail over security. And they are rubbed almost close to the side of the tram. Movement means a high threat of collision.

And then the most interesting thing: how to configure the system so that the tram can move in such difficult conditions and so that the system does not work continuously during braking. Therefore, you first need to start moving, this will change the reaction of drivers, and then again you need to evaluate this reaction. This is one of the most non-trivial tasks.

image

The second feature is idiots on the roads, which fly off to the side from minor ones and rush on.

Since the real situation during rush hours is often accompanied by repeated violations of the rules by others, the driver often wants to take control so that the tram does not become cautious. We made the “10 seconds of freedom” button when the system shuts down and restrictions are removed. Most often, the driver uses it now at “locked” intersections. The frequency of use of this button is one of the KPIs by which we evaluate the quality of the system.

How to come to a brighter future


The first level is avoiding collisions, assisting the driver, responding to vivid threats faster than him.

At the second level, there was a driver control mode: it switches the arrow along the route, controls the speed of the sections (the driver can now enter the turn faster than safe speed or can stick the 15th running into ice), but the tram will not accelerate faster than the profile speed.

We can implement further levels in closed areas. The tram moves under human control. A co-pilot takes him along the route, takes into account the situation on the roads, at times better protects the tram and others.

And a brighter future is a fully autonomous tram.

Installation was tested on Vityaz and Bogatyr (this tram is not yet found on the roads). Everything is controlled by them through the CAN interface module. No non-native modules in the cab, all through system interfaces. Equipment - a camera on glass.

image
We use POE cameras for industrial vision with different lenses (you need to look far ahead, wide on the sides). The radar is more interesting: the review is 160 degrees, the range is 150 meters, the rain breaks through. We also need a high-precision GNSS sensor for navigation and integration units with the tram itself. Processing takes place on site, the 65 TFLOPS processing unit (looks like a large heatsink with ports). As you can see, this is not a Belarus tractor for autopilot, you can get more expensive equipment here.

Moscow and other cities are moving towards light rail transport. There are approximately 300 developed tram networks in the world. They are best made in China, Russia, Turkey, and are developing in Europe. In China, the development of the tram network is crazy because it runs on the regional budget, and the metro on the federal one. The budgets of the region are more easily allocated and more active than federal ones. The other day we signed a contract with FiTSCO - the largest Chinese rail operator , we are launching our system in Shanghai, and we will have the first case of a Chinese tram.

In Russia - 1,500 trams. In the world - 30 thousand. Production - only 2,500–3,000 per year, that is, now we will equip existing trams with the second or third level, and then we will move on to integration with manufacturers.



You already know about our co-pilots for combines and tractors , now you know about the co-pilot of the tram. There is still a shunting locomotive. All this equipment in terms of market size is a third of the automobile. Not a miserable interest, but a third. And in this market, only we are now with such technologies. So it’s very interesting not just to observe how the world is changing, but to change it.

I’ll also note from surprises that this very radar is of our development. Because at first we needed to collect data from a low level to obtain more accurate models in neural networks, and then somehow word for word we put together the first prototype - and away we go. About how to collect the radar, I think it is worth telling separately if you are interested.

More articles:


All Articles