Get best of the week

Neural networks in the production of dentures

Replacing a tooth with an implant or installing a crown is a painful and expensive procedure. One of the most difficult parts in the restoration is the design of the prosthesis in the CAD system, which dental technicians are engaged in. Each crown is individually designed for the patient and his jaw in 8-10 minutes. At the same time, each technician has his own subjective vision of what a good tooth crown is, and the quality assessment of the same crown for different specialists of the same level can vary from “good” to “possible and better”.


Therefore, it is not surprising that in dentistry the goal was to remove the human factor and add automation. This can be done using neural networks. They have now advanced so much that they can recognize objects, find criminals in the crowd, draw pictures by sketch , and replace the faces of actors in films, for example, DiCaprio with Burunov in the movie The Great Gatsby. They also help to cope with teeth, and how it happened, Stanislav Shushkevich will tell.

Stanislav Shushkevich - works for Adalisk Service. This is the contractor for the largest US denture manufacturer. The tasks of the contractor include automation of the production of crowns, bridges, implants. Stanislav teaches deep neural networks. At Saint HighLoad ++ 2019, Stanislav made a report in which he described in detail and in detail how the company uses deep training to automate input data classification, design generation and crown modeling automation.

This article is based on the transcript of his report, from it you will learn how to apply deep learning and design generation in the production of dentures, how to stabilize quality, automate various stages of production using neural networks, gradually reduce the human factor and reduce the average time by several times. which the dental technician spends on crowns and implants.

Note. The group, in which Stanislav works, collaborates with the Berkeley Institute (USA). They worked together to develop deep neural networks to automate the design of dental implants. Based on the results of this work, the researchers published a scientific article , but the report is more interesting.

Dental terminology


To immerse you in the course of affairs, I will tell about dental terminology. A person has 32 teeth, but more often 28, because wisdom teeth often do not grow or grow sick and are removed. There is a continuous numbering from 1 to 32, but it is more convenient to divide the jaw into 4 numbered quadrants of 8 teeth. Therefore, we say that a person has 32 teeth - from 11 to 48. The first digit is the number of the quadrant, the second is the number of the tooth in the quadrant.


Standard tooth numbering scheme.

After mirror reflection, the teeth on the left side of the jaw are similar to the teeth on the right, but the upper ones are not similar to the lower ones. Therefore, when we say that we need to generate a tooth, we usually mean 16 teeth - 32 in half is just 16.

The shape of a person’s teeth cannot determine sex. Therefore, when excavation finds teeth, it is not clear whose they are: men or women. At the same time, you can understand from the jaws of the horse whether it is a mare or a horse - the horse has fangs, but the mare does not. Such "dental" facts always amuse me.

The stump of the 36th tooth is marked in blue - the diseased tooth was sawn off, the stump remained. In English, the tooth stump is called "die." The edge of the stump is highlighted in red, it is called "margin". When the dentist puts the crown on the tooth, it is important that the edge of the crown almost coincides with the margin, with a margin for “glue”, otherwise there will be an ugly step.


Three-dimensional scan of the impression of the lower jaw.

For us, the technician made a 3D model of a beautiful crown.



Grooves on top - anatomy of the crown. They are located not randomly, but by their typical tooth number. It can be described for a long time, but there is a primary anatomy, a secondary, orthodontic cross.

The jaw antagonist is suitable from above, and if the tooth is upper, the jaw is antagonist from below. The upward direction, which is called the occlusal direction , is also important to us . When an “occlusal snapshot of a tooth” appears in the text, it simply means a snapshot from above.

Travel crowns


How do dental crowns do? Those who set themselves dental crowns roughly represent. I will tell the rest of the general scheme.

Dentist - Dental Technician - Dentist. You come to the dentist, open your mouth, he says repair. Files a tooth and gives a bite to the plate. It looks like plasticine on a substrate - an imprint of the upper and lower jaw remains on it. The dentist sends this cast to the laboratory to the dental technician.

A dental technician sits in a laboratory and never communicates with real patients. He builds tooth models in a CAD / CAM system on a computer. From the finished 3D model of the tooth, he makes a real restoration and sends it back to the dentist. Further, the dentist is already installing the restoration in place.
Our application of power is at the center of this scheme.
Dentist - Dental Technician + ML - Dentist . Now the dental technician works in a program that is similar to AutoCAD. But ideally, he should do this: a case comes in the form of a mold, a technician scans it and receives a ready proposal for the crown. Our ML group works with dental technicians and over the years has collected 5 million cases and 150 TB of data that are on Amazon - there are plenty to choose from.

The technician takes 10 minutes to prepare a 3D image of the crown , while he thinks about many parameters:

  • natural form;
  • antagonist;
  • tight contacts;
  • margin line;
  • jaw segmentation;
  • deep anatomy;
  • directions;
  • arch;
  • filling in papers and others.

Parameters


The naturalness of the form . This is the first thing a dental technician thinks about - for me this is the most mysterious moment. When I show the technique of two teeth and he says which one is good, I ask why he decided so:

- I don’t know. I just see that this is a natural tooth, but it is not.
- Are there any formal criteria? An automated design system needs formal metrics.
“I don't know any formal metrics.” I just see that this is a good tooth, and this one is bad.

This is called the "natural form."

In this case, the technician additionally takes care that the crown does not intersect with the antagonist and sits well on the "edge" of the stump, about tight contact with neighbors, sets directions and segments the jaw.

There are several good solutions for any case. Different techniques for the same jaw will create several different good teeth.

Why not make it simpler: take a regular tooth crown, “play around” with the shape of the geometric algorithm and insert it into the jaw? This geometric task is simple in imagination, but in practice it is not so simple.

Variability . In the picture below are three images of crowns.

The first on the left is the perfect human tooth. On average, the gap between the antagonist and the tooth stump is so small that the crown is flattened. The distance between this tooth and the neighbor on the left was large enough, but it was necessary that the crown touched the neighboring tooth, so an influx appeared on the left. In the right figure everything seems normal, but the line of the “edge” of the stump is not a circle, but a complex spatial figure. For geometric methods, all this is difficult due to the large variability of the crowns.

Attention to detail. Another important detail is the characteristic size of the simulated features. The size of the 36th tooth (from the bottom of the lower jaw) is approximately 8 mm, and 200 microns is the size of the anatomy only. This means that if normal anatomy is needed, then a model accuracy of at least 100 microns is required. It is cut approximately with an accuracy of 20 microns. We divide 100 microns into 8 mm - we get an accuracy of about 1%.

All this creates a problem - the technician spends a lot of time on many factors. We decided to simplify its work and increase efficiency - more crowns per unit time. We tried several options and chose neural networks, which simplified the work of the technician.

Solution methods


CAD / CAM . Each “dental” company creates its own system. Glidewell Dental, Invisalign, Sirona - all have or are developing their own CAD / CAM systems. The systems look like AutoCAD or KOMPAS-3D: you have a standard object, you can pull, stroke, rotate it. Now, technicians work this way and spend 10 minutes to produce a normal crown.

ML - harmonic functions. This is another way - relatively simple machine learning. For example, if you look at the crown from above, it will look like a circle with grooves, and then you can try to decompose it into harmonic functions. But we tried - with our conditions for accuracy, this does not work out.

Therefore, in the end, we turned to deep networksto generate three-dimensional objects. I will tell two stories that are related to our choice. The first is propaedeutic . This is a training or educational story about jaw segmentation. The second is about crown generation .

Jaw segmentation


Segmentation of the jaw is a painting of each tooth and gum with its own color (for example, on an occlusal view).



Segmentation is important. For example, the jaw came in the form of a 3D model, and on it a sawn tooth stump. In order for the program to understand where to put the tooth, it should at least be positioned relative to the stump. In this case, a person usually sits and with a mouse clicks it and its neighbors, and only then does the program understand where to put the tooth. With automatic segmentation, everything would be much more interesting - it would automatically indicate where to put the tooth.
.
Neural network training requires data. There are open datasets for segmenting faces, people's poses, letters - people enjoy attending conferences and sharing successes in training with each other. But finding a dataset for tooth segmentation is difficult.

There are companies that have datasets with already segmented teeth. But they, of course, do not sell or give them away. This is the know-how and great value of every company.

Fortunately, we had a Watershed geometric algorithm , which I will discuss later. He can segment, but with an efficiency of 30% - normal segmentation in one out of three cases. We segmented the algorithm with 15 thousand cases, and then manually filtered. After filtering, 5 thousand good cases remained on which we trained SegNet.

Note. SegNet is the standard network for segmentation. A specific network is not so important. It is important what to do, in what sequence and where to get the data.


This is how jaw segmentation with 12 teeth looks.

Everything seems to be fine - different beautiful colors, the teeth practically do not “leak”. But it’s of little use to segment only those cases that we have already learned to segment geometrically. We would like to learn to segment all phase space. In the case of a strong correlation between the geometric and the neural network algorithm, the network will mainly segment only those cases that are already well segmented.

So the main question is: is there a correlation between what Watershed and SegNet are doing? To answer this question, you need to know how these algorithms work.

How Watershed and SegNet Work


Watershed works on a 3D surface — these are “valleys” separated by ridges with great spatial curvature. When we move on a 3D surface, in some places there is a sharp bend, for example, where two teeth mate or one tooth enters the gum. In these places “ridges” arise. Watershed "pours water" and it covers the valleys, but does not cross the "ridges".

The algorithm does not work well where the spatial curvature is broken. For example, there are two teeth with a common tangent. They were scanned so that two teeth smoothly pass into each other. The algorithm will paint two teeth with one color.

SegNet works like any segmented network. He knows what can be approximately obtained inside - a segmented picture usually looks like 14 circles located in an arc, and around the gum. SegNet is prone to errors: when the circles are irregular in shape, the patient has not 14 teeth, but 12 or the tooth falls out of the arch - the arc on which the teeth are located. The picture shows just 12 teeth, it was hard, but the algorithm did it.

It seems that Watershed and SegNet do not correlate with each other, and everything is relatively normal.

Subtotals


Optimization is required for the end user . We could spend a lot of time in advance to get rid of the correlation, think about it, take action. But even without this, SegNet, when training on selected cases, yielded about 90% of the correct segmentations - a network of 9 out of 10 jaws segments perfectly.

Manual labor helps a lot. You will find out your data and select what you need.

We pass to the main dish.

Crown Generation


To generate the crown, we chose this scheme: we take the occlusal view and from it a monochrome map of depths - depth map.


Crown with deep anatomy (cross) and with secondary anatomy.

Data sources


The first source of data is natural teeth . There are many casts of the jaws, we now know how to segment them: we cut out natural teeth and train on them.

We took them, but it turned out badly. Natural teeth are too varied . Ordinary human teeth are not very beautiful, even among young people. I want the crown to be nicer.

The second data source is ready-made crowns.. We already have 5 million cases of 150 TB. They are stored in the Amazon cloud. Of the 5 million cases, we select those that have completed the techniques, and train on them. But it didn’t work out either. We carefully looked at our training set and found that from half to two-thirds of the finished crowns can be done better. This mainly concerned the depth of anatomy on the finished crowns - the grooves were not sufficiently expressed.

This was an unpleasant discovery, because we took the results of specialists who should be imitated. But we already knew what to do in such cases.

We took 10 thousand cases and manually broke them into good and bad. Received 5 thousand good on which you can learn. But experimentally we knew that for training it takes from 10 to 15 thousand good cases. To get them, you need to manually sort 30 thousand cases per tooth - this is too much. Therefore, we trained a simple auxiliary network which showed teeth, and it separated the good from the bad.



The figure shows that the three upper teeth have a deep anatomy - the cross is clearly visible, and on the lower depressions. On the last (leftmost) there is no anatomy at all. The technician “licked” this crown so that the anatomy disappeared.

With the help of an auxiliary network, we can filter very large volumes, and receive 10-20 thousand cases per tooth.

Generation Technical Details


Fines. The first thing that occurred to me was to take the generating network, show her the upper picture of the jaw with a stump of the tooth, require her to draw the lower one, and impose a fine L1. But the theory says that it won’t work, and that’s why.


The jaw with die above is with a sawed tooth, in the middle it is an antagonist, and the crown below, which should be created as a result of the network.

We have already said that there are many good solutions for the same input. If you simply impose a fine L1, then you will fine the network for the fact that it could not guess the image of the crown in the head of the technician at the moment when he designed it. He could make such a crown, or he could make another, also good. A fine for another good crown is not necessary.
L1's “bare” penalty is a bad idea.
Discriminator. It’s a good idea to train a discriminator who says “Good” to all good crowns and “Bad” to bad crowns. It will take into account the complex surface of good crowns (surface in the space of crowns). In addition, GAN, as it turned out, crushes high-frequency noise.

Our Loss is as follows: Loss = D_GAN + L1 + AntagonistIntersectionPenalty.

D_GAN- takes into account the complex surface of possible good decisions. AntagonistIntersectionPenaltyadded so that the tooth does not intersect with the antagonist.

It is important that L1 suddenly appears. You just read that it should spoil something, but if you add it in a limited way, then it does not spoil. The reason is that the GAN is quite unstable during training, and L1 reports that the tooth in any case looks like a white spot in the middle of the frame. At the initial stage of training, it stabilizes - everything converges better, looks smooth and neat.

Technical Note We fought for a long time, trying to train one net for all back teeth or one net for a couple of teeth. But we came to the conclusion that you need to train your network for every tooth, for every small localized piece of data. We have such an opportunity.
One tooth - one network.
This is important - this is your trade-off between the stability of the solution and the resources used. If you spend an extra 200 MB of video memory, then (usually) nothing will happen. But you will keep a separate network for each tooth and train it as needed.

Subtotals


We again used auxiliary networks and manual labor :

  • sorted by hand, broke 10 thousand cases into “good and bad”;
  • on the results of manual sorting we learned the auxiliary network;
  • bulk sifted unmarked cases.

Hooray, we went into production!


After all that we did, we went into production. This is not quite a real production - the Research & Production group, but it creates 100 crowns per day.

Note. At the time of publication of the article, generating networks are already included in this product .

The group has been working since May - several thousand crowns were made by GANs. A technician presses a button and a crown image is generated in 20 seconds. The technician checks the correctness of the form, usually tightens the contacts, and sends it to a drink.

We received a substantial gain in time. The crown is prepared in 8-10 minutes, and with the involvement of the GAN - in 4 minutes. The GAN covers 80% of the cases - if the technician does not like what the GAN suggested, he models the crown with his hands and spends 8 minutes.

The lessons


Optimization for the end user . In the process, it’s useful to think about measures, metrics, loss function and correlations. But you did everything right if you brought the work to the end and got the expected result.

Use of auxiliary networks.

Manual labor. There is a saying in ML: " Know your details." Using manual labor, you will find out what is in your data. Manual labor is usually rewarded because you “feed” your network with exactly what you need.

The balance of "quality - resources . " If possible, shift towards quality. Don't be greedy - add as many networks as you need.

Saint HighLoad++. 6 7 - , , . — , Call for Papers . , - , , .

Source: https://habr.com/ru/post/undefined/

More articles:


All Articles