📷 👠 🆗 How we made the autopilot for the combine harvester on video analytics the first in the world 🌿 💢 👲🏾

That's the whole set, if there is a CAN bus.

Just five years ago there were no normally functioning neural networks for the same determination of obstacles and stalk edges, therefore there was no video analytics either. There were “blind” GPS methods, which in practice turned out to be far from the best and greatly dampened the prestige of automation in agriculture. In five years, we believe that all combines will be automated precisely by visual autopilots to look from the cockpit and to the sides and control all aspects of harvesting.

We are in the moment when we already have ready-made technologies, they are well tested, cheap and have annual operating experience, and large combine harvesters look at them with interest. Most likely, it will be like with car radios: first they are put into cars, and then the cars come with already built-in ones. So we are now modifying old combines, but we want to take a place in the ecosystem and put the complex on all new ones.

Such a project could start in our country, in Brazil, and even in a couple of countries due to the specifics of the market. We need a country with agriculture, developers inside, inefficient harvesting (that is, a clear pain to save) and a new fleet of combines. We were lucky with everything, including the park: after the USSR, everything fell apart, and now we are seeing cars about four to five years ago in the main farms.

There are 350 thousand combines in Russia, and another 35 thousand are delivered annually. This is not a car market, of course, but by making the right decision now, while no one has come there, you can get it all.

But let’s tell you better how it works and how we modify combines in Russia.

How the working day of the combine operator with our Agro Pilot goes

The combine operator - the operator or machine operator - arrives at seven in the morning on the field. If this is not the first day, then the car is already directly in the middle of crops. Settings are made for the next section, warming up. At nine in the morning the dew leaves the fields, you can start. You need to finish in the dark when the humidity rises again. You can’t work in the rain. In about two weeks, you need to have time to collect all the grain before it pours and spills out onto the ground: the farther, the greater the losses. You cannot start earlier: the grain has not yet ripened. Therefore, it is precisely at this peak of loads that any simple, any error or accident is very expensive.

The change lasts up to 14 hours of monotonous work. The operator needs:

, . . .
: , 20–22 , . , . .
— .
— .
.

Here, my colleague Eduard spoke in more detail about the features of why, in the end, operators focus only on taxiing , leave the default settings in the collection mechanism and often skip the power line support or tractor in front and crash into the 10th shift.

A short summary: a person can either steer, or monitor the quality of cleaning. It turns out badly to steer, because the combine has a 12 meter dimension along the header, and often large mowed stripes remain due to imperfect turns.

If you remove the load on taxiing, then suddenly the grain harvest increases miraculously. In our practice, last year from the same site - by 3% due to the fact that the operator has the opportunity to more accurately look at what is happening ahead. And about 3-5% more - due to retention of capture without "belly buttons" and large mowed parts. Plus no accidents.

In general, this is a very necessary thing that would have long been implemented if there had been a technical possibility.

Technical opportunity appeared

Here's what autopilot does:

That is:

The combine is driving as it should according to the characteristics of the growth of the current culture and how the combines went to it.
It allows you to work together a mixed fleet: both for those cars that have an autopilot, and with manual control. Autopilot makes no difference.
Very clearly keeps the distance between the aisles, ensuring a minimum of mowing.
He searches for obstacles, classifies them and decides what to do: go round, slow down or, while there is time, warn the operator.
Support for optimal speed for a specific situation. There is such a feature: for the harvester to clean cleanly, you need to keep the interval from 6 to 6.5 kilometers per hour for, say, wheat on a specific model. If we cross this line, then the mass will be larger, the scourges will stop knocking out all the grains, and further the losses increase exponentially from speed: seven kilometers - 0.2%, eight - 0.5%, and so on.

Agrotechnical evaluation of one of the combines according to the results of laboratory field tests

How speed affects collection performance



	1			2			3

, /	4,1	5,4	6,3	4,5	5,6	7	2,8	3,6	4,3
,	4,8
,	14			23			6

:
— ,	—	13	13	22	22	22	5	5	5
	—	1,4	1,42	1,65	1,86	3,37	1,35	1,47	1,19
— ,	—	9,6	9,5	8	8,6	14,8	12,2	12,1	9,4

— , %
20%, , %	—	0,13	0,17	0,27	0,35	0,36	0,56	0,62	0,74
:
—	—	0,08	0,11	0,2	0,24	0,25	0,29	0,32	0,31
—	—	0,05	0,06	0,07	0,11	0,11	0,24	0,27	0,4
—	—	0	0	0	0	0	0,03	0,03	0,03
, /	10,7	15,3	16,1	2,7	3,4	4,5	3,6	4,1	6,1
, /	5,28	6,75	7,72	2,09	2,68	4,84	4	5,3	7
, /	3,85	4,15	5,37	2,23	2,87	5,99	5	7	8,7
, , %,	0,83	0,93	1,01	0,39	0,44	0,5	0,33	0,49	0,53
:	0,24	0,37	0,42	0,27	0,32	0,37	0,26	0,42	0,45
—	0,24	0,37	0,42	0,27	0,32	0,37	0,26	0,42	0,45
—	0,43	0,41	0,45	0			0
— -	—	0,003	—
— , %	0,16	0,15	0,14	0,12	0,12	0,13	0,06	0,07	0,08
, %:
—	1,56	1,5	1,41	1,18	1,2	1,27	0,62	0,6911,27	0,83
—	0,42	0,5	0,57	0,88	0,89	1,09	1,32	98,04	1,72
— ()	98,02	98	98,02	97,94	97,91	97,64	98,06		97,45
,	1052	1048	1063

The main function of our equipment now is not auto-driving, but the exclusion of the human factor from the operator’s work. The basic set of our equipment helps to eliminate the recession of reapers and unnecessary runs.

What's in the stuffing

1. 2 MP camera on the mirror bracket. We focus only on the camera: this is the main sensor. Nothing more is required.

2. Cab display: this is the interface for the operator. Through it, warnings and settings are made.

3. Control unit under the cab: there is NVIDIA TX2. The main algorithm stack spins there, the video is processed, and from there commands to the CAN bus are issued.

Block 340x290x60 mm, 40 watts.

The module for issuing commands is connected to the CAN bus or other system input / output of the combine. There are a couple of ambushes, in particular, with the fact that not everywhere is possible, and not everywhere hydraulic control is possible through this interface.

In the case of the harvester of the last five years of production, this is almost always everything: a box with brains, a camera, a display in the cab.

The wheel rotation sensor is needed in case the combine is old or without preparation, and we cannot get this data from the system bus. We need data for odometry (speed and angle of rotation of the wheels):

The metering pump in the hydraulic system is needed to directly control the hydraulics by our co-pilot:

The first ambush with the CAN bus is that there is not always clear documentation. In theory, there are other options for receiving signals from the bus, but in practice in Russia these are very rare cases. In general, at best there is some kind of system API and a cable hole. In a simple case, we contact the manufacturer of this hole and ask for a description of the protocol. After a couple of weeks of negotiations, they understand who we are and what we need, and send. Not everyone perceives this simply, but large producers are partly happy with us because they know that last year we put our solutions in a number of farms and large agricultural complexes, which caused a bit of noise in the media.

And they often immediately ask if our technologies can be integrated into the harvester of the future. This is a separate story, but as a result of the first part, we have a protocol, and everything is fine.

In an ideal world, that would be the end. Unfortunately, sometimes there are still cases when there is no documentation or the manufacturer fails to get a clear description. In this case, we connect and reverse the bus. Of course, there is a J1939 protocol that manufacturers should follow, but not everyone does it. Usually you connect with the debugging utility, you get all the packets with variables and just sit with the operator clicking on the buttons. I went - somewhere zero became a positive number. Accelerated - the number has increased. Slowed down - decreased. Yeah, the speed is. And so - all day. Next, you need to catch the correlation and select the conversion factors. Once we caught a very unpleasant bug with what worked on the test site, but not in the field. The harvester in peace mode sent some packets,and with the reaper and reel turned on, he went into a special combat mode and began to send his special combat messages in the same captured packets. We had to collect data signs in hellish ways. As it turned out, the header gave its information normally to the same packets into the same variables, and the bus designer optimized all this wildly so as not to set new variables. Probably, he participates in the 256b-intro competition in his spare time.

There were surprises on hydraulics too. Although Danfoss is in the partners, although they helped everyone as much as possible and gave detailed schemes, everything can be checked only by starting the combine. Somewhere, the components slightly stood at the border, so the system for the customer came to harvest (and this is only one of the cases). That is, it was necessary to mount it all at night, start up, check and, if at least something is wrong, remove it so that the combine could be used further in the morning. We arrived at eight in the evening and began to shaman with hydraulics. We finished the first installation at four in the morning. They did not know whether it would start or not. If it does not work out, then it will be necessary to disassemble and return to its normal state. Started up - it works. Steers as it should. This is the coolest hello world in our life. I didn’t imagine working with neural networks, but where without it.

You can add a cellular modem to send telemetry to the kit. However, the kit does not need GPS. This is a huge plus. For normal GPS driving to work normally, you need to map in advance, set up an RTK station for corrections, or buy a signal package and so on. And still press buttons and menus, and machine operators really dislike user interfaces. We have a camera-box-boarded-let's go. It is not necessary to map the field to cut it into pens. You need to drive to the field. The robot will say: “Oh, cheers, field!” And just go.

Why only a driving camera

In 2014, we took a scientific grant for a prototype of an integrated agricultural enterprise management system using AI. We plunged into the business of agricultural producers and looked at the most obvious points of automation. Russia is a very risky crop (one crop per year, and five in Brazil). You cultivate a field for a whole year: expensive seed (this is the most expensive), then you sprinkle with expensive chemicals (second expense item), conduct a cycle of work on preparing soil and crops. This is a whole year from March to autumn. And only in the fall you have two weeks when you need to harvest. If something goes wrong, then a plus or minus a day can be 10% yield loss.

The harvester did not come out, the harvester washed down, crookedly squinted - broke along the road. Everything, simple for hours and days.

It was with the recognition of obstacles that we started. Immediately a lidar came up, but it is expensive, so the camera. Once a monocamera - you need to immediately recognize images. Because you need not only to see, but also to understand what it is, what size it is, how you behave, what to expect. We either stand still or move forward, but you need to know the relative distance to the object and still remember about the header 7.5 or 9 meters in different directions (and this is not the maximum, often on running models there are also 12-13 meters). All the same, on such dimensions, it is possible to accidentally grind a tractor operator or a refueling machine out of a small need at a U-turn.

Neural networks do an excellent job. But still, since we have a monocamera, you can fool the algorithm with a 1:40 combine model in rare cases (because you can also get geometry from a monocamera in motion). But such rarely come across in the field.

In corn (three meter stalk), it is important to look for pillars, for example. Since we had a definition that here is culture and what is an obstacle, then we could further distinguish where what culture is.

And this is the second level autopilot:

It was very difficult with the culture itself. At the first iterations of the trips, we came to the field for a new culture or a specific one that looked specific. Agronomist poorly watched, for example, did not spray on time. Low crop with weeds - the neural network does not eat. Once it was that we came to the farm for testing. And there is a sharply different climate, and barley does not look at all like in the reference book of the agronomist. And in our segmentation does not recognize the mowed and beveled parts correctly. There were few data for the training sample, but we ourselves shot them. All day until the evening we rode through the fields with a machine operator, took a new sample. They sat down at the hotel at night with their staff to mark out these data. Put on retraining network. The next day, everything drove into a new field.

This is how you can add the target sample:

According to Eduard, who explains this to farmers, people themselves perfectly understand that cleaning is the main point of automation. There feeds the day a year. And they are trying with all their might to exclude human errors. In the end, we agree on a “look." Before the season comes our service group or dealers, install everything. Then carry out a calibration check. Or, before the harvesting period, they are immediately calibrated, it will take a day and a half.

If it’s interesting, then I can tell you later about the features of recognition of everything that is found in the fields, or about how we wonderfully collect training samples, because the finished ones are disgusting and not suitable for Russian conditions. This is also due to the fact that we are the first to do anything with this topic around the world, and therefore there are no established practices yet.

PS If your agronomist is not on Habré, and he is interested, then you can find contacts here: promo.cognitivepilot.com and substantively discuss for which combine what specific set of equipment is needed, how much it costs, and how you can quickly see it- to experience.

How we made the autopilot for the combine harvester on video analytics the first in the world

Technical opportunity appeared

What's in the stuffing

Why only a driving camera

More articles: