🎅🏻 🏑 🍅 As we wrote the world's coolest autopilot for a shunting locomotive 👩‍👧‍👧 🌍 🐉

One of the earliest prototypes used for tests.

I must say right away: he is the coolest because he is the only autopilot of the third level brought to trial operation. And he was the only one brought to trial operation because without the experience of autopiloting trams and something else, it simply makes no sense to get into this market. There are a lot of diesel locomotives, the task is interesting and important for production, but it does not pay off as a separate one. We know about the developments on this topic at NIIAS and Siemens, but we don’t know that their trams traveled somewhere in an urban environment, and locomotives transported real goods.

Since we already have a lot of different developments and solutions with unmanned trams in Russia and China, we decided to experiment with one large enterprise with a large fleet of shunting diesel locomotives used to deliver raw materials to workshops.

There, the problem is that the movement of the locomotive is regulated by many signals, the positions of people and infrastructure, as well as the teams of the dispatcher. The driver should remain extremely attentive throughout the shift (approximately 12 hours), including at night. As a result, sooner or later, he either misses something and has an accident, or knocks someone down. This is life, transport injuries happen, but specifically in these situations, you can afford to put radars on diesel locomotives, because not just one diesel locomotive gets up, but a whole large enterprise. For a long time. Collision avoidance and autopilot can greatly reduce the load on a person in the cockpit, and then the production will not get up.

The module in the picture is one of the earliest prototypes of the camera unit that we started with. From that moment he underwent significant changes, but it is always interesting to see how it all began. Now I’ll tell you how robots are generally able to navigate at stations, because the task is actually non-trivial.

Our autopilot ensures safe operation in difficult weather and climatic conditions and in any light condition.

Here's what is put on the locomotive:

The current version in the hardware of the camera unit is very different from the first prototypes.

Three cameras are needed because this design can move quite quickly, the stopping distance, unlike the tram, is large (trams get almost in place in an emergency), and the load is heavy behind. Therefore, it is necessary to predict the situation in advance. Very often it is necessary to read the indications of traffic lights in such stories:

Therefore, there are three cameras:

Here is what they see:

Plus radar.

A setup is needed quite expensive, about 15 thousand dollars. This will not be put on a car, because the complex should not cost more than a car. But it’s reasonable to deploy such a hardware-software complex on diesel locomotives, and it pays off. And the task is different. At the last conference, Google showed its full set of sensors, which, in his opinion, should be installed on every unmanned vehicle, where the price is much more than 100 thousand dollars. Both they and we have convergent duplication: if one of the subsystems fails, then it will be possible to get to the depot (service) for the rest.

Why started in Russia

Because we are in TOP-3 in agricultural machinery, in TOP-5 in railway, and we have one of the most developed tram networks in the world (but beyond the top ten, compare at least with China). The world market of tractors, combines, trams and diesel locomotives is one third of the automobile. Third. And we have an ideal country for the development of all this, also because in the 90s they all collapsed and didn’t buy, and a lot of the equipment is now almost fresh, that is, there are almost no generations before last.

In the world - 300 thousand locomotives. In our country - 50 thousand locomotives from Russian Railways. Of these, 15 thousand are shunting. In addition, there are 10 thousand locomotives of private owners. These are most often large plants - metallurgical plants. 250 locomotives - on Novolipetsk, for example, where we conducted tests. They solve a specific problem: the continuity of the process of delivery of raw materials to production or shipment of finished products to the main railway. There are chemical, metallurgical and mining enterprises with such tasks.

Less than 1,000 diesel locomotives are produced per year. This is aftermarket - you need to deal with the current fleet. It is necessary to equip it.

And we can do this very well.

The number of accidents in closed areas is greater than it seems from the outside. They are almost always without sacrifices, but with material damage (arrow cut, the train went off the rails), and the rhythm of production is always broken while repair is underway.

The task is to make the L3 - an unmanned locomotive.

Decision

Tram operating time allows you to keep the path well:

And it’s good to see the obstacles:

But just getting and rolling the code onto the locomotive will not work. The fact is that if the tram route is quite simple (two-track with a minimum of branches), then the locomotive has much higher complexity of stations, and you need to navigate very well in a dense cluster of tracks.

Plus shunting diesel locomotives - they do not travel all over the country, but move along their closed routes. The first thing we did was to start making track maps as an auxiliary data source based on plans and satellite images (we must admit that it turned out to be much faster and more reliable with satellite images). That is, they trained a neural network to highlight paths, but so far we are adjusting the result by hand. The following schemes were obtained:

On board there is a high-precision GPS + GLONASS (also expensive, but here you can), a visual orientation scheme and a radar. This means that you can attach in coordinates approximately to a section of the route, and due to the recognition of surrounding objects, “find out” the exact place.

It is convenient when there is a satellite map.

Therefore, the next thing we did was to start teaching diesel locomotives to stations. In fact, at each passage along the route, the locomotive remembers the surrounding objects and recognizes them in order to divide them into permanent and temporary (it’s strange to navigate by a canister or a person, but by the totality of poles, traffic lights and arrows it’s quite possible). Then he projects onto the map. This is one of the data sources, and it makes it possible to greatly improve the accuracy of decision-making by autopilot. The storage algorithm is approximately the same as with FaceID: new successful data is added to the training set.

Naturally, we needed to mark up datasets manually at the beginning. Not the most pleasant and fast work, but very necessary. It was possible to collect datasets only at the facility, because much inside was nothing like commercial banks.

One of the features of the stations is bright light and poor conditions for recognition. Here necessarily need the radar to search for pedestrians, video analytics by itself does not pull:

Here's how you can draw out the bad scenes (arbitrary obstacles on the tracks, the distance estimation to objects) at the expense of stereo vision, which creates even more data to the low-level definition of environmental picture:

Since radar - of our own design, we are able to receive low-level data from all sensors and pull out more correlations from neural networks directly on the fly from them than usual. Including from the original radar scan. This also turned out to be very important for the project.

Of course, we also use forecasts of the movement of people (more details are in the last post about the tram). But here everything is simple, people almost do not go and do not try to jump under a diesel locomotive in attempts to get into it. Here are the obstacle avoidance:

This is how the position of the arrows is detected: we wait for the arrow in navigation, look for the arrow, switch to the far camera for a closer look, select the segment with the arrow in the frame dragging it into the neuron, we get the result of the state of the arrow.

What is the difference between the tasks of a shunting diesel locomotive from the usual?

Shunting - these are such devices:

Here it is more beautiful:

Such a locomotive is almost constantly occupied by the fact that it is approaching and docking with the train.

Features of the movement is that he pulls and pushes. It moves in two directions, that is, half a day goes against the main view from the cab.

There are a lot of restrictions on maneuvers; there are own traffic lights, besides the usual ones.

If there are few arrows on ordinary lines, then the shunting inside the station usually has a whole network of switches. GPS + RTK corrections are not enough at the stations, because there are a lot of different iron around. Kalman filtering does not guarantee path preservation. We constantly have one and a half meters - the next way, and we can position ourselves there, this leads to algorithm errors. We used a particle filter for low-level data - it allowed 10 times to improve positioning accuracy. Particle filtering is difficult in computing (this can be circumvented), and we may not explicitly specify a motion model. But you can give him data from all sensors: this is odometry, poles at the station, traffic lights and arrows, we are binding to a virtual map. Works under bridges. Distribution in front of the pillarsshooter and traffic lights at the station allows you to give a much more accurate assessment of their own position than GPS. In the future we plan to abandon RTK corrections altogether, GPS will be needed only for an initial assessment of the situation. Next is the correlation of the pillars on the map and on the camera / radar.

Traffic lights are more complex than on a tram. It is necessary to distinguish between traffic signals. On the lines there is an ALSN, and for shunting tasks, the signal along the rail is not duplicated inside the station. Need a night mode, this is when traffic lights illuminate the camera. In the daytime mode, they are sometimes pale too. For this, they came up with a filter system. It is necessary to recognize for 300 meters in order to have time to stop the composition.

With a prepared and marked-up satellite map, it turns out well, but the autopilot can learn after five or six trips to the new station itself, therefore, if the office does not prepare the map, you will need to travel empty at the new stations. On the other hand, if the card is out of date, then our robot will not get lost. It is important that this consumes a lot of resources, because you need to not just look at the objects, but analyze the type of each. The transfer officer can become a permanent object, because he is always in one place, but if he is assigned a class of a person, then the robot understands that it makes no sense to put it on the map.

The driver needs an assistant to survey 180 degrees in the direction of travel, because the locomotive has a structure with the engine compartment in front, and it closes the view to the driver on everything on the left. And either the driver must ride with an assistant (usually one is not allowed), or a review and autopilot system is needed.

After testing and certification, our system can be dispensed with without an assistant on the site. The system will not fall asleep.

Here is one of the best illustrations that happens. BelAZ collided with a diesel locomotive. The locomotive lost, and BelAZ's suspension flew off and the beam broke. Now you know, the whale wins or the elephant.

It was quite difficult with braking. In the case of a tram, we flexibly control the process and can do partial braking. In the case of a train - only to a complete stop, otherwise a train break may occur. The tram could slow down and control the time before the collision. This is due to its braking system and fixed weight. Each time the train is different, there are up to 100 cars, and the air brake is removed for a few seconds. The big problem is that if we control the brakes of the train, then we need to control the pressure throughout the train, which was slightly out of control for the autopilot.

What exactly does the system do?

Now it initiates auto-braking at a traffic light that prohibits the position of a woolly arrow, when approaching a train, it is faster than tolerance or when there are obstacles on the tracks (including a person). We calculate the forecast of the collision, the trajectory of objects, the speed of the train. If a person, according to his intentions, crosses the paths at the same time when the train goes there, then first a warning, if time is short, we initiate inhibition.

General principle:

The test base was in Vologda, because there they take security very seriously. A few years ago, the locomotive driver fell asleep and drove into the tanks through a prohibition signal. As a result, the car turned over, a bunch of tanks went off the rails, it's good that they were empty. There are photographs, they are very impressive.

If the tanks weren’t empty, then there wouldn’t have been any photographs, just everything would have taken off into the air. A tired driver will get a prohibition light more often than it seems from the outside. A very likely case is not gouging, but a misunderstanding of the dispatcher’s command on the walkie-talkie (you can ride under the blue if the dispatcher’s permission does so). But if you do not look at the arrow, it will be hacked. Nothing will happen to the current squad, but the next one following it in the opposite direction may go off the rails.

This is how it looks from the driver’s cab:

As you can see, we are rightly moving towards the fact that the old ChMEZiki (in some regions they are also called Chmukhs) will be very perfect unmanned vehicles. Well, or with an assistant co-pilot at least. It already has and works.

Afterword

In December 2019, we completed a project with Russian Railways to create a hardware-software complex for assisting the driver, presented a prototype shunting locomotive with the possibility of autonomous control. The first results of the work were shown to the management of Russian Railways in July last year and were highly appreciated. Today, under the agreement, Cognitive Rail Way Pilot intelligent systems are installed on ten shunting locomotives of the ChMEZ brand.

Head of Russian Railways Oleg Belozerov with road managers and the Governor of the Tula region Alexei Dyumin take jobs with Cognitive Pilot specialists as part of the XV Assembly of Railway Chiefs at the Schekino station of the Tula Railway.

The Cognitive Pilot team after successfully demonstrating the results of a joint project to the management of Russian Railways and the Tula region at the Schekino station of the Tula railway.

As we wrote the world's coolest autopilot for a shunting locomotive

Why started in Russia

Decision

What is the difference between the tasks of a shunting diesel locomotive from the usual?

What exactly does the system do?

Afterword

More articles: