Research has established that a large percentage of dental cavities escape identification in routine dental examinations, even when such examinations include dental x-rays. Certain types of cavities, like occlusal cavities, appear to be easier to find through a normal clinical examination or X-ray review, whereas diagnosing other types of cavities, such as cavities below the surface of the tooth, interproximal cavities, or root cavities, is often not as reliable.
At ParallelDots, Inc., we took the challenge to find these dental anomalies with human-level accuracy and build a reliable diagnostic tool for dentists. In this blog post, we will discuss the recent study we did for our automated cavity detection system, putting our system against three practicing dentists from North American clinics. We found that our system had a higher agreement (F Score) with clinically verified ground truth than all three individually (the difference between system’s F Score and average F-Score of the dentists is over 17%). Our system has higher sensitivity with respect to dentists individually and hence can be used as a tool to ease the work of dentists by suggesting them possible cavities they can then verify and treat. A breakdown of metrics in our test is given in the table below. Please note that the metrics Recall and Precision are names that the ML community generally uses for Sensitivity and Positive Predictive Value.
Performance comparison between our system and testing dentists.
Our paper detailing the experiments performed has been accepted at the NIPS 2017 Workshop on Machine Learning for Health being held on the theme, "What Parts of Healthcare Are Ripe for Disruption by Machine Learning Right Now?" NIPS (Neural Information Processing Systems) is among the topmost machine learning conferences globally and has two tracks of papers and multiple focused workshops.
How Does the Automated Cavity Detection Technology Work?
The first step is to gather a dataset to train the AI algorithm on. We obtained IRB approval and collected data from multiple clinics across the US, where dentists marked clinically verified cavities on their X-ray archives. 3,000 radiographs with cavities annotated were collected as part of this exercise. The data is anonymized as soon as the clinician uploads it to our HIPAA-compliant server and the dataset thus is ready to be used by our data science team.
The next step for the data science team is to decide:
How to model this as a machine learning problem.
How to calculate the accuracy of the machine learning system thus created.
Generally, the above processes go hand-in-hand (for example, you would measure accuracy for something you would model as a pixel-wise classification task using the dice coefficient for an object detection task as Intersection over Union or for a classification task as accuracy). In this case, however, the problem arises due to the following complications:
- Dental cavities are irregularly shaped; thus, there is no way for dentists to mark them exactly. At best we can get approximate polygon annotations for dental cavities. Although modeling the cavity detection problem as Dense Classification task is pretty obvious, a dice score will not be a fair measure of accuracy on such approximate annotations.
- While average IoU score might be a good measure, it's hard to visualize how effective the model is just by this number in the real world. Hence, we put a more practical criterion for the model to be evaluated. We evaluate the model as if it were a search engine to find cavities. A search engine is evaluated based on measures of precision, recall, and F Score. Precision denotes the number of correct searches among total results returned by the cavity search. Out of the total cavities present, the number of cavities that the system can detect is called recall.
A radiograph containing cavities and the output processed by our algorithm is shown below.
Detecting cavities from an input radiograph.
With this research, we are aiming to assist dentists in analyzing and detecting dental problems. The automated cavity detection system can provide solid assistance to dentists and make their work easier, faster, and more accurate so that they can effectively concentrate on rectifying dental problems much faster.