previous chapters

34. How to determine the level of quality available to man

Suppose you are working on a medical imaging application that should do automatic x-ray diagnostics. The mistake of an ordinary person without a medical education, with the exception of some basic training, is about 15%. The error of a novice doctor is about 10%. An experienced doctor is mistaken in 5% of cases. The error of a small team of doctors studying and discussing each image does not exceed 2%. Which of these figures should be taken as the "level of human quality"?

In this case, I would take 2% for the level of quality available to the person and establish the corresponding optimal error value. It also makes sense to set 2% as the desired error level for our system, since this error value meets all three criteria described in the previous chapter for systems that allow you to compare the quality of the algorithm with the quality of the task performed by a person:

• Ease of tagging data : you can use a team of doctors to tag data with an accuracy of 98% (2% error)
• Error analysis using human intuition : When discussing X-rays with a team of doctors, you can rely on their intuition when looking for ways to improve quality
• Using a person’s task completion level to establish the optimal error level as well as to determine the achievable “desired error level” of the system : It is advisable to use 2% as an estimate of the optimal error level. The optimal error level can even be less than 2%, but it certainly cannot be higher due to the fact that such an error level corresponds to the quality of diagnostics available to specialists and it makes no sense to set the error level to 5% or 10% for the automatic system, since we know for sure that we will achieve a deliberately higher level of quality.

, , . , .

40%, , ( 10%) ( 5%) . 10%, , 2% , .

35. ,

. , 10% . , 8% . - , 33 ?

, , , . , , , .

, :

1. , ,
2. , ,
3. « » .

, , , , , . , , , .

There are many important machine learning applications in which algorithms are superior to humans. For example, cars can better predict movie ratings, travel time by car, loan repayment. In cases where it is difficult for people to find examples in which the algorithm is clearly mistaken, only a limited number of methods can be applied to improve quality. Therefore, when working on a system that has already surpassed humans, progress usually progresses more slowly than in cases where algorithms have yet to reach the human level.

continuation