Get best of the week

How we created a prototype robot merchandiser and what's next



Coronavirus continues its deadly spread throughout the world and in our country. We have been working from home for almost two months, like all IT employees around the world, and, on the one hand, we are more and more sad about our good old openspace and the opportunity to discuss all tasks and problems in a living, and not crazy useful, but still soulless zoom (we unfortunately did not have time to rent a California llama for all our meet-ups :)). On the other hand, we are thinking more and more about automation and robotization of all processes, including the processes of creating the analytical data that we and our system work with.

In more detail, we recall our first robot merchandiser, which we made as a prototype 2 years ago and are more and more thinking (and not only thinking!) About the second version, which can be put into commercial production and operation.

In this article I want to tell you what happened from this idea, what turned out badly or did not work out at all, and what we are going to do now in order to bring this idea to real application in real stores. For those who are interested, welcome to cat.

A little background: our company (ShelfMatch Robotics) began to automate merchandising about 5 years ago. Automation in this case means that in one way or another we get photos of goods shelves in stores (in pharmacies, at gas stations, etc.) made by merchandisers, recognize all the goods on them and issue the necessary analytics to customers.

As soon as we got a little deeper into it, we immediately faced one of the main and rather complex problems - for the most part, merchandisers take bad or very bad photos. It is clear that this issue can be resolved in various traditional ways, such as staff training, issuing good devices, etc., but for many reasons (I will leave them aside, unimportant), these methods do not always work, or rather, they do not always work well.

The second, no less important reason why automation of analytics inside the store makes sense is that stores lose money every second due to various display errors, such as out-of-stock or failure to observe the planogram. And they lose a lot of money, according to analysts at bossa nova robotics out-of-stock, leading to a loss of revenue of 0.5 trillion. dollars a year all over the world. In general, for the store, the introduction of “extra eyes” that will scan the entire space of the trading floor and warehouse is extremely important and poorly implemented at the moment.

We at that moment immediately came up with another, more interesting, in our opinion, decision - why not delegate taking photos to someone who, in essence, does everything ALWAYS exactly according to the instructions and almost never makes mistakes (when taking into account accurate and high-quality instructions) . You probably already guessed that we are talking about a robot-merchandisers.

I remember when we considered all the craziest ideas, starting from robotic drones that will fly around the store and ending with robotic automatic cameras that will travel along mono-rails on the ceiling of the store. The first thing that we saw and really inspired us was the robots of bossa nova robotics, which cut across American stores (for example, Walmart), giving customers the opportunity to touch the near future a little. As a result, we nevertheless came to the conclusion that we need to create a traditional robot on a "chassis-truck" such as a robot vacuum cleaner and equip it with all kinds of cameras and sensors necessary for navigation and photographing shelves.

Formulation of the problem:


So, the statement of the problem is quite simple: we need a prototype robot that can independently (while under the supervision of the operator, but still without controlling the joystick) travel along the shelves of the store, photograph all these shelves and transfer the photos to the server for further processing. The robot should look, if not attractive, then at least not frightening, so that shop visitors do not shy away from it. This is important, since we planned to conduct real experiments and measurements in real open stores and no one was going to close them for the period of our manipulations.

The first version of the robot was named after the commander of replicants from the good old, I hope everyone is familiar with the familiar and beloved film of Ridley Scott Blade Runner - Roy (as we will continue to call him).

What Roy consisted of


Chassis:

  • Swiss motors maxon - Swiss because their Chinese cheap comrades for some reason very quickly burned out in our hands
  • 4 wheels of the omni-wheels type - at that time it seemed to us that we needed exactly omni-wheels so that the robot could quickly and without unnecessary movements change the direction of movement
  • base made of aluminum and plastic
  • motor controller
  • battery
  • NUC with ROS on top of Ubuntu inside for navigation
  • Lidar RPLidar A2 for building maps, navigation and localization

Torso (tower):

  • aluminum profile frame
  • plastic finish
  • plastic mounts for cameras printed on a 3D printer
  • 4 USB Bustler Cameras
  • NUC for taking pictures and sending photos to the server via WiFi
  • 3D camera of depth like Intel IntelliVision, which we still could not connect
  • LCD screen for debugging and displaying service and advertising information

It was not difficult to assemble the trolley and body, we spent several months on it. In the assembly and adjustment of the trolley we were assisted by several former employees of the TsNII-RTK.

How it all worked and what came of it


Everyone who is familiar with working in ROS can skip this chapter; they are unlikely to find something unique or original in it, but those who are “off topic” will be interested. I’ll tell you right away how all this happened in a real store (we were then tested in the still live and healthy SPAR in Kupchino):

  1. We come to the store, find the rack we need (the prototype was tested on the product category Tea, then another 4-5 product categories were photographed), we find an unoccupied outlet (this, by the way, was quite difficult), and unpack the entire belongings.
  2. Roy ( — , , ).

  3. WiFi- .
  4. Ubuntu+ROS , Game-Pad .
  5. : ROS, Teleop ( ), Gmapping ( ), Rviz ( Roy).
  6. — , Roy , ( , Roy «» «»).
  7. After the map is built, save it, hammer in the coordinates of the survey points (the beginning of the rack 1 is point “A”, the end of the rack 1 is point “B”, the rotation is 90 degrees, the end of the rack 2 is point “C”. We basically removed the rack in the shape of the letter "G")

Next, run the photography script (everything is already without a gamepad, real “magic”):

  1. Roy goes to the given position - point "A".
  2. Roy is deployed in the correct way (with cameras to the shelf), the degrees of rotation are also set in the script.
  3. Roy begins to drive slowly along the rack, shooting several times per second with each camera and saving the photo to the NUC. In this case, the cameras are synchronized and take photos at the same time.
  4. At the same moment, another script starts asynchronously sending photos via WiFi to the server.
  5. Roy drives along one rack (point “B”), stops the cameras for the duration of the turn, makes a 90-degree turn, turns on the cameras again.
  6. It drives along the 2nd rack to point “C”.
  7. Turns off the cameras.
  8. Returns to the starting position (point "A").

Further work is already directly on the server:

  1. At the moment when all the photos are transferred to the server, the server starts gluing photos and the process of recognizing goods on the shelf
  2. When all processes are completed, in the web interface there are racks with goods marked with goods markup + all the necessary analytics (the share of goods on the shelf, out-of-stock, comparison with the planogram, etc.)



Summing up what we managed to find out


  • perhaps the most important point - we can make a prototype of the robot on our own, there is nothing unrealistic about it, the whole system (collecting photos + navigation + data transfer + data processing + data visualization) works, but this is only a prototype
  • an interesting and important observation - visitors to the store are not afraid and do not “shy away” the robot - grandmothers just don’t notice it, children admire it, others are very interested in “what kind of thing is it and why is it necessary”, merchandisers worry because it is their direct competitor
  • the question of the selection of spare parts must be approached wisely - given the cost, availability, quality

Roy flaws identified after prototype creation and testing


The biggest problem we had with building a map and the associated navigation problem

  • the map was not always built exactly
  • very often it was built poorly from the 1st or 2nd time, it took half an hour to go to build a map
  • Roy sometimes "lost" even on a good map (this was immediately clear since he started spinning in one place and didn’t go anywhere else, he had to turn on the gamepad and switch to manual control)

We had such hypotheses about this problem:

  • our omni wheels sometimes slipped over the smooth tiles of the store, this could knock down odometer parameters
  • Roy had too few "eyes", maybe one lidar was not enough for him and it was necessary to add other sensors
  • maybe our lidar was not the most correct and accurate or was poorly tuned, maybe the problem was in it, especially considering the complexity of the task (a small distance to the shelf, very long racks)

Other disadvantages:
  • — (+ ), 30 60 , Roy 2
  • ( ), ( )
  • Roy , , - (QR- , bluetooth- ..)
  • () ( ) , ,
  • ( , , , )
  • ( , ) ( 10 )
  • Roy , , , ( , « »)
  • Roy ( 3-4 ),

2 (Zhora Leon)


The pandemic of the coronavirus has shown that in some situations the use of a robot is not only desirable, but even necessary. If we are in a situation where it is better not to leave the house at all, robots, delivery men, cleaners, and analysts could help us in solving a large number of important tasks. If you look at our Roy or his brothers from this side, the project looks even more interesting - for example, the idea of ​​re-equipping a robot-marchandiser into a robot-disinfector or a robot that monitors the observance of social distance and unobtrusively warns those who crowd in one place or queuing up.

The platform on the basis of which we want to build a new version can not only scan shelves, but also recognize the faces of visitors. Other interesting features of such devices could be:

  • targeted advertising on LCD screens: from the series I see a woman with a baby - I offer diapers, I see pensioners - I offer goods on stocks;
  • help customers find products
  • monitoring of shoplifters on the basis of persons
  • warehouse inventory, assistance to the logistics department in procurement planning
  • etc. right up to putting goods on shelves

Summing up, we can say that we would like to see in the 2nd version of the prototype:

  • industrial "sexy" design, providing for a bunch of all kinds of "tricks" like a device for communicating with the outside world (speakers, microphone, screens, cameras, indicator of emotions, etc.)
  • all processes need to be significantly accelerated
  • the new device should be easy to scale and relatively cheap
  • need accuracy in navigation and localization
  • need global navigation throughout the store
  • +

P.S. - ( ), , .

P.P.S. , , .

More articles:


All Articles