Get best of the week

Warehouse robots using AI to sort items are ready to go

Berkeley startup Covariant stepped out of the shadows and believes his robots are ready to enter the world




In the summer of 2018, a small startup from Berkeley, developing robots, faced a difficult task. Knapp, a major supplier of warehouse logistics technology, was looking for a new AI-controlled robotic arm that could capture as many different types of objects as possible. Every week, eight consecutive weeks, the company sent a start-up list of increasingly complex items - opaque boxes, transparent boxes, packages of medicines, socks - covering the product range of the company's customers. The start-up bought such items, and then a week later sent a video in which their roboruk transferred items from one gray basket to another.

At the end of the quest, the Knapp leadership was defeated. For six or seven years they have been unsuccessfully giving similar tasks to many startups, and they expected failure this time too. But instead, in each video, the startup’s robot assistant shifted every item with perfect accuracy and the right speed.

“With each next product, we expected failure as the task became increasingly complex,” said Peter Pachwein, vice president of innovation at Knapp, headquartered in Austria. “However, it turned out that they were successful, and everything worked. We have never seen such a quality AI job. ”


Covariant is now out of the shadows and announces a collaboration with Knapp. Its algorithms already work in Knapp robots in the warehouses of two company customers. One of them belongs to the German electric goods manufacturer Obeta, and robots have been working there since September. Startup co-founders say Covariant is close to closing another deal with another industrial giant manufacturing robots.

This news symbolizes a change in the current state of AI robotics. Such systems were limited to an artificial academic environment. But now Covariant claims that its system can generalize work to the difficulties associated with the real world, and is ready to storm warehouses.

Warehouses have tasks for two equipment options - for machines with legs moving boxes here and there, and for machines with hands lifting objects and putting them in the right place. Robots have long been present in warehouses, but their successes were mainly limited by the automation of the first option. “People rarely move in a modern warehouse,” says Peter Chen, co-founder and director of Covariant. “The transfer of things between fixed points is a problem that mechatronics does very well .”


Robotic arm in a Covariant office

But not only the right hardware is required for hand automation. Technology needs to quickly adapt to a wide range of shapes and sizes of products with an ever-changing orientation. A traditional robotic arm can be programmed to perform the exact same movements over and over again, however, it will fail as soon as it encounters a deviation. She needs AI to “see” and adjust, or she will not be able to cope with the developing environment. “Intelligence is needed for dexterity,” Chen says.

Over the past few years, research laboratories have achieved unprecedented success in combining AI and robotics, achieving similar dexterity, however, transferring these achievements to the real world is a completely different task. In laboratories, an accuracy of 60-70% is acceptable; this is not enough in production. And even with a 90% accuracy, a robotic arm would be a “value-loss offer,” says Peter Abbil, co-founder and chief scientist at Covariant.

Abbil and Chen estimate that in order to actually recapture the investment, the robot must achieve an accuracy of 99-99.5%. Only then can he work without frequent human intervention and the risk of slowing down the conveyor. However, only recent progress in deep learning, and in particular in reinforced learning, has allowed this level of accuracy to be achieved.


The Covariant office is located off the coast of San Francisco Bay, next to a dilapidated parking lot between rows of unmarked buildings. Inside, several industrial robots and “co-bots,” collaboration robots designed to work safely with humans, are trained to work with all possible products.

Covariant team members regularly run to the local store for various random items. Things range from lotions in bottles to packaged clothes and erasers in transparent boxes. The team is especially interested in things that can confuse the robot: reflective metal surfaces, transparent plastic, easily deformable surfaces like clothes or packets of chips that will look different on the camera every time.

Over each robot there are several cameras that work with his eyes. Visual data and sensory data from the body of the robot enter the algorithm that controls its movements. Basically, the robot learns from a combination of simulation and reinforcement. The first is that a person manually controls the robot, lifting various objects. Then he records and analyzes the sequence of movements in order to understand how to generalize this behavior. The second is that the robot does millions of repetitions of trial and error. Each time, trying to take a thing, the robot does it a little differently. Then he writes down which attempts ended with a faster and more accurate lifting of the subject, and which failed to constantly improve his efficiency.

Since the algorithm is ultimately trained, the Covariant software platform, Covariant Brain, is independent of hardware. There are a dozen robots of various models in the office, and the robot working at Obeta uses Knapp hardware.





For an hour, I watched three different robots confidently pick up completely different items from the store. In seconds, the algorithm analyzes their location, calculates the angle of attack, adjusts the sequence of movements and reaches out to take them with the suction cup. It moves with confidence and accuracy, and changes the speed of work depending on the fragility of the subject. With medications wrapped in foil, he manages more gently so as not to deform the packaging and not crumple the medicine. During one particularly impressive demonstration of the work, the robot redirected the air flow so as to blow off the bag uncomfortably pressed against the wall to the center, so that it was easier to lift.

Knapp Pachwein says that since the company switched to the Covariant platform, its robots have moved from the ability to lift 10-15% of items from the Obeta range to the ability to lift about 95% of items. The last 5% are such fragile things as glass, which only people can still work with. “And this is not a problem,” adds Pachwein. - In the future, 10 robots and one person will be a typical warehouse device. That is the plan. ” Thanks to the collaboration, Knapp will distribute its robots with Covariant software to the warehouses of its customers for several years.

Although the result is impressive from a technical point of view, it raises questions about how such robots will affect automation. Pachwein admits that he expects that in the next five years, hundreds or thousands of robots will begin to perform tasks traditionally solved by humans. However, he says, people still no longer want to do this job. In Europe, companies often find it difficult to find enough people to work in warehouses. “This is the kind of feedback we get from all customers,” he says. “They don’t find employees, and they need more automation.”

To date, Covariant has already received $ 27 million from investors, including AI luminaries such as Turing Prize winners Joffrey Hinton and Ian Lekun. The startup wants to deal not only with lifting objects, but also with the whole spectrum of warehouse operations, from unloading trucks to packing boxes and sorting on racks. The startup also has ideas about moving outside of warehouses and entering other industries.

But the ultimate goal of Abbil is even higher: “The long-term idea of ​​the company is to solve all the problems in the field of AI-robotization.”

More articles:


All Articles