Get best of the week

Automated COVID-19 Detector from Fluorography Images



In this guide, you will learn how to automatically detect COVID-19 in a specially selected dataset using Keras, TensorFlow, and deep learning.

Like most people in the world right now, I am sincerely concerned about COVID-19. I notice that I began to constantly analyze my personal health and suspect that I have already become infected.

The more I worry about this, the more a painful combination of real symptoms with hypochondria manifests itself:

  • I woke up in the morning, I feel a little overwhelmed.
  • When I got out of bed, I already had a leak from my nose (although it is now known that a runny nose is not a symptom of COVID-19).
  • While he reached the bathroom to take a napkin, he also coughed.

At first I did not attach much importance to this - I am allergic to pollen, and due to warm weather on the east coast, spring came early this year. Most likely, my allergy just broke out.

But the symptoms did not improve all day.

And here I am sitting here, writing this manual with a thermometer in my mouth; and looking down shows 37.4 ° C. The temperature is higher than my usual 36.3 ° C. Everything above 37.2 ° C, I already consider a slight fever.

Cough and a little fever? It could be COVID-19 ... or just my allergy.

It is impossible to find out without a test, and it is precisely this “ignorance” that makes this situation so dangerous for mental balance.

For people there is nothing more terrible than the unknown.

Despite all the fears, I try to get rid of them in a rational way. I'm over thirty, I'm in great shape, my immune system is very strong. I’ll go to quarantine (just in case), I’ll rest and recover normally. COVID-19 does not scare me in terms of my personal health (at least I keep saying this to myself).

However, I worry about my elderly relatives, including those who have chronic illnesses, and those in a nursing home or hospital. They are vulnerable, and it will be truly sad if we lose them due to this infection.

In order not to sit idly by and not succumb to weakness (be it allergy, COVID-19 or just anxiety), I decided to do what I do best -serve the entire community of machine vision and deep learning by writing code, conducting experiments, and teaching others how to use computer vision and deep learning in practical, real-world applications .

However, to be honest, this is not the most scientific article I have ever written. In fact, far from the most. The methods and data sets used do not deserve publication. But they can be the starting point for those who want to help.

But I want to do everything in my power. This article is simply my way to mentally overcome difficult times and at the same time help others in a similar situation.

I hope you understand me.

In today's lesson we will do the following:

  1. We will collect a set of open data from fluorograms of patients with a positive test for COVID-19.
  2. We collect a selection of fluorograms of healthy patients.
  3. We will train the convolutional neural network to automatically detect COVID-19 in the images from the created set.
  4. We will evaluate the results obtained from an educational point of view.

Disclaimer: I have already hinted, but I’ll say it bluntly. The methods and techniques in this article are for educational purposes only. This is not a real scientific study, and it will not be published in a journal. The article is intended for readers who are interested in (1) computer vision / deep learning and want to learn using practical methods, and (2) are concerned about current events. I beg you to relate to this in this way.

COVID-19 on fluorograms


In the first part, we discuss how COVID-19 can be detected on chest radiographs.

Then consider our data set.

Then I'll show you how to train the deep learning model using Keras and TensorFlow to define COVID-19 in our image dataset.

My task is to inspire you and show that the study of computer vision / deep learning, and then applying this knowledge in the medical field can have a big impact on the world.

Simply put: you do not need a medical degree to contribute to medicine. Deep learning specialists who work closely with doctors and healthcare providers can solve complex problems, save lives, and make the world a better place.

I hope this guide inspires you to do just that.

But with all this said, scientific journals and expert assessment systems are full of materials with forecasting models of COVID-19 of dubious quality. Please do not take the code / model from this article and do not send it to the journal or put it in the public domain on scientific sites - you will only add some noise.

In addition, if you intend to use this article (or any other COVID-19 article from the Internet) in your own research, be sure to check the TRIPOD rules for creating forecast reporting models .

As you probably know, the use of AI in medicine leads to very real consequences . Publish or use such models only after close consultation with a medical expert.


Fig. 1. An example of a fluorogram from a patient with a positive test for COVID-19. Using Keras and TensorFlow in such images, it is possible to train the classifier to detect COVID-19. COVID-19

tests are currently difficult to find - they simply are not enough and they cannot be made fast enough, which causes panic.

When there is a panic, some vile people take advantage of this and sell fake test kits COVID-19 , finding victims on social networks and instant messengers .

Given that the number of COVID-19 test kits is limited, you should rely on other diagnostic methods.

Doctors often rely on chest x-rays (commonly called fluorograms) to diagnose pneumonia, pneumonia, abscesses, and / or enlarged lymph nodes.

COVID-19 attacks the epithelium of the respiratory tract, and X-rays help to examine the patient’s lung condition. X-ray machines are installed in most hospitals, so it is logical to use fluorography to test COVID-19 without special test kits. The disadvantage is that such an analysis requires a specialist in radiology and takes considerable time. Therefore, the development of an automated system will be very valuable to save the valuable time of medical professionals around the world.

Note. Some more recent publications suggest that computed tomography is better for diagnosing COVID-19, but in this lesson we only work with fluorograms.

Data set



Fig. 2. On the left are positive samples (from infected patients), and on the right are negative. These images are used to train a model to automatically predict the presence of a disease.

Our data set was compiled by Dr. Joseph Cohen , a graduate student at the University of Montreal. In early March, he began collecting fluorograms of patients with COVID-19, MERS, SARS and SARS and publishing them in the repository on GitHub .

We need to download a file from the repositorymetadata.csvand select the lines from it where it is indicated

  1. Positive result on COVID-19.
  2. Type of lungs 'posterior-anterior' (PA). As far as I understand, this is the type of image used in “healthy” images.

A total of 25 fluorograms were found with positive cases of COVID-19 (Fig. 2, left).

For healthy patients, I used the Cuggle dataset (pneumonia) and selected 25 images of healthy patients (Fig. 2, right). This data set has a number of problems, namely noisy / incorrect labels, but it will do for a conceptual check of the COVID-19 automatic detector.

Project structure


In the Downloads section of my website, you can enter your email address and find out a link to the archive with code, scripts and other files for this project. Extract the files and you will see the directory structure:

$ tree --dirsfirst --filelimit 10
.
├── dataset
│   ├── covid [25 entries]
│   └── normal [25 entries]
├── build_covid_dataset.py
├── sample_kaggle_dataset.py
├── train_covid19.py
├── plot.png
└── covid19.model
3 directories, 5 files

Fluorograms are in a directory dataset/where two data classes are divided into directories covid/and normal/.

Both scripts are presented for building a data set, but we will not consider them today. Instead, consider a script train_covid19.pythat teaches our COVID-19 detector.

Implementation


Let's move on to setting up a convolutional neural network. Open the file train_covid19.pyand paste the following code:

# import the necessary packages
from tensorflow.keras.preprocessing.image import ImageDataGenerator
from tensorflow.keras.applications import VGG16
from tensorflow.keras.layers import AveragePooling2D
from tensorflow.keras.layers import Dropout
from tensorflow.keras.layers import Flatten
from tensorflow.keras.layers import Dense
from tensorflow.keras.layers import Input
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.utils import to_categorical
from sklearn.preprocessing import LabelBinarizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix
from imutils import paths
import matplotlib.pyplot as plt
import numpy as np
import argparse
import cv2
import os

This script uses a number of deep learning libraries from TensorFlow 2.0 and Keras. In addition, we use scikit-learn , the standard Python library for machine learning, matplotlib for graphing, and OpenCV for loading and processing images.

To install TensorFlow 2.0 (including the appropriate scikit-learn, OpenCV, and matplotlib libraries), just follow my instructions for Ubuntu or macOS .

Next, we will analyze the command line arguments and initialize the hyperparameters:

# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--dataset", required=True,
	help="path to input dataset")
ap.add_argument("-p", "--plot", type=str, default="plot.png",
	help="path to output loss/accuracy plot")
ap.add_argument("-m", "--model", type=str, default="covid19.model",
	help="path to output loss/accuracy plot")
args = vars(ap.parse_args())
# initialize the initial learning rate, number of epochs to train for,
# and batch size
INIT_LR = 1e-3
EPOCHS = 25
BS = 8

Three command line arguments :

  • --dataset: path to the input dataset
  • --plot: An optional path to the exit schedule for your learning history. Unless otherwise specified, the default isplot.png
  • --model: optional path to the output model; defaultcovid19.model

The next lines initialize the initial learning speed, the number of eras, and the hyperparameters of the packet size.

Now we are ready to download and pre-process the fluorograms:

# grab the list of images in our dataset directory, then initialize
# the list of data (i.e., images) and class images
print("[INFO] loading images...")
imagePaths = list(paths.list_images(args["dataset"]))
data = []
labels = []
# loop over the image paths
for imagePath in imagePaths:
	# extract the class label from the filename
	label = imagePath.split(os.path.sep)[-2]
	# load the image, swap color channels, and resize it to be a fixed
	# 224x224 pixels while ignoring aspect ratio
	image = cv2.imread(imagePath)
	image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
	image = cv2.resize(image, (224, 224))
	# update the data and labels lists, respectively
	data.append(image)
	labels.append(label)
# convert the data and labels to NumPy arrays while scaling the pixel
# intensities to the range [0, 1]
data = np.array(data) / 255.0
labels = np.array(labels)

To load data, we capture all the paths to the images in the catalog --dataset. Then for each imagePath:

  • We retrieve the class label ( covidor normal) from the path .
  • We load imageand pre-process it, converting it to RGB and resizing to 224 × 224 pixels, so that it is ready for the neural network.
  • Updating lists dataand labels.

Then scale the pixel intensity to the range [0, 1] and convert the data and labels to the NumPy array format.

Next, we perform direct encoding labelswith dividing the data into sets for training / testing:

# perform one-hot encoding on the labels
lb = LabelBinarizer()
labels = lb.fit_transform(labels)
labels = to_categorical(labels)
# partition the data into training and testing splits using 80% of
# the data for training and the remaining 20% for testing
(trainX, testX, trainY, testY) = train_test_split(data, labels,
	test_size=0.20, stratify=labels, random_state=42)
# initialize the training data augmentation object
trainAug = ImageDataGenerator(
	rotation_range=15,
	fill_mode="nearest")

Direct encoding means that the data is converted into the following format:

[[0. 1.]
 [0. 1.]
 [0. 1.]
 ...
 [1. 0.]
 [1. 0.]
 [1. 0.]]

Each encoded label consists of an array with two elements: 'hot' (1) or 'not' (0).

Then the following lines separate our data, leaving 80% for training and 20% for testing.

To make sure the model is generalized, we perform augmentation of the data by setting the image to randomly rotate 15 ° clockwise or counterclockwise.

The last three lines initialize the data augmentation generator object.

Now initialize our VGGNet model and fine-tune it :

# load the VGG16 network, ensuring the head FC layer sets are left
# off
baseModel = VGG16(weights="imagenet", include_top=False,
	input_tensor=Input(shape=(224, 224, 3)))
# construct the head of the model that will be placed on top of the
# the base model
headModel = baseModel.output
headModel = AveragePooling2D(pool_size=(4, 4))(headModel)
headModel = Flatten(name="flatten")(headModel)
headModel = Dense(64, activation="relu")(headModel)
headModel = Dropout(0.5)(headModel)
headModel = Dense(2, activation="softmax")(headModel)
# place the head FC model on top of the base model (this will become
# the actual model we will train)
model = Model(inputs=baseModel.input, outputs=headModel)
# loop over all layers in the base model and freeze them so they will
# *not* be updated during the first training process
for layer in baseModel.layers:
	layer.trainable = False

The first two lines create an instance of the VGG16 network with weights previously trained on ImageNet without touching the fully connected layer (FC).

Then we build a new fully connected layer, consisting of layers POOL => FC = SOFTMAX, and add it on top of VGG16.

Then we freeze the weights CONVfor VGG16, so only the FC layer will be trained . This completes the setup.

Now we are ready to compile and train our deep learning model for COVID-19:

# compile our model
print("[INFO] compiling model...")
opt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)
model.compile(loss="binary_crossentropy", optimizer=opt,
	metrics=["accuracy"])
# train the head of the network
print("[INFO] training head...")
H = model.fit_generator(
	trainAug.flow(trainX, trainY, batch_size=BS),
	steps_per_epoch=len(trainX) // BS,
	validation_data=(testX, testY),
	validation_steps=len(testX) // BS,
	epochs=EPOCHS)

The first lines compile the neural network with the option to reduce learning speed and the optimizer Adam. Given that this is a two-class problem, we use loss "binary_crossentropy"rather than cross entropy.

To start the learning process, we call the fit_generator method in Keras , passing the fluorograms through our data augmentation object.

We evaluate the model:

# make predictions on the testing set
print("[INFO] evaluating network...")
predIdxs = model.predict(testX, batch_size=BS)
# for each image in the testing set we need to find the index of the
# label with corresponding largest predicted probability
predIdxs = np.argmax(predIdxs, axis=1)
# show a nicely formatted classification report
print(classification_report(testY.argmax(axis=1), predIdxs,
	target_names=lb.classes_))

To evaluate, we first make forecasts on the test set, preserving the forecast indices. Then we create and display a classification report using scikit-learn, where there is such an auxiliary function.

Next, we calculate the matrix of inaccuracies for further statistical evaluation:

# compute the confusion matrix and and use it to derive the raw
# accuracy, sensitivity, and specificity
cm = confusion_matrix(testY.argmax(axis=1), predIdxs)
total = sum(sum(cm))
acc = (cm[0, 0] + cm[1, 1]) / total
sensitivity = cm[0, 0] / (cm[0, 0] + cm[0, 1])
specificity = cm[1, 1] / (cm[1, 0] + cm[1, 1])
# show the confusion matrix, accuracy, sensitivity, and specificity
print(cm)
print("acc: {:.4f}".format(acc))
print("sensitivity: {:.4f}".format(sensitivity))
print("specificity: {:.4f}".format(specificity))

Here we do the following:

  • We generate a matrix of inaccuracies.
  • With its help, we calculate accuracy, sensitivity and specificity by printing all these metrics.

For self-testing, we generate a graph with a history of training accuracy and losses, output it to a graphic file:

# plot the training loss and accuracy
N = EPOCHS
plt.style.use("ggplot")
plt.figure()
plt.plot(np.arange(0, N), H.history["loss"], label="train_loss")
plt.plot(np.arange(0, N), H.history["val_loss"], label="val_loss")
plt.plot(np.arange(0, N), H.history["accuracy"], label="train_acc")
plt.plot(np.arange(0, N), H.history["val_accuracy"], label="val_acc")
plt.title("Training Loss and Accuracy on COVID-19 Dataset")
plt.xlabel("Epoch #")
plt.ylabel("Loss/Accuracy")
plt.legend(loc="lower left")
plt.savefig(args["plot"])

Finally, save the classifier model tf.kerasto disk:

# serialize the model to disk
print("[INFO] saving COVID-19 detector model...")
model.save(args["model"], save_format="h5")

Detector training


After downloading all the files from the archive with the sources and scripts mentioned above, open the console and run the following command to train the detector:

$ python train_covid19.py --dataset dataset
[INFO] loading images...
[INFO] compiling model...
[INFO] training head...
Epoch 1/25
5/5 [==============================] - 20s 4s/step - loss: 0.7169 - accuracy: 0.6000 - val_loss: 0.6590 - val_accuracy: 0.5000
Epoch 2/25
5/5 [==============================] - 0s 86ms/step - loss: 0.8088 - accuracy: 0.4250 - val_loss: 0.6112 - val_accuracy: 0.9000
Epoch 3/25
5/5 [==============================] - 0s 99ms/step - loss: 0.6809 - accuracy: 0.5500 - val_loss: 0.6054 - val_accuracy: 0.5000
Epoch 4/25
5/5 [==============================] - 1s 100ms/step - loss: 0.6723 - accuracy: 0.6000 - val_loss: 0.5771 - val_accuracy: 0.6000
...
Epoch 22/25
5/5 [==============================] - 0s 99ms/step - loss: 0.3271 - accuracy: 0.9250 - val_loss: 0.2902 - val_accuracy: 0.9000
Epoch 23/25
5/5 [==============================] - 0s 99ms/step - loss: 0.3634 - accuracy: 0.9250 - val_loss: 0.2690 - val_accuracy: 0.9000
Epoch 24/25
5/5 [==============================] - 27s 5s/step - loss: 0.3175 - accuracy: 0.9250 - val_loss: 0.2395 - val_accuracy: 0.9000
Epoch 25/25
5/5 [==============================] - 1s 101ms/step - loss: 0.3655 - accuracy: 0.8250 - val_loss: 0.2522 - val_accuracy: 0.9000
[INFO] evaluating network...
              precision    recall  f1-score   support
       covid       0.83      1.00      0.91         5
      normal       1.00      0.80      0.89         5
    accuracy                           0.90        10
   macro avg       0.92      0.90      0.90        10
weighted avg       0.92      0.90      0.90        10
[[5 0]
 [1 4]]
acc: 0.9000
sensitivity: 1.0000
specificity: 0.8000
[INFO] saving COVID-19 detector model...

Automatic diagnosis of COVID-19 according to the results of fluorography


: , «» COVID-19. . , , , ( , ). , / COVID-19.

As can be seen from the above results, our automatic detector has an accuracy of about 90-92% on an existing set of samples. This rating is based on image analysis only. During training, no more data was used, including geographical location, population density, etc.

The sensitivity was 100%, and the specificity was 80%, which means:

  • Truly infected patients are recognized in 100% of cases.
  • Healthy patients are recognized as healthy with a probability of 80%.

As the training history graph shows, our network is not too retrained, despite a very limited data set: The



ability to recognize COVID-19 with absolute accuracy is great, although the rate of false positives is a little worrying, because many healthy people will be sent to quarantine with really infected patients.

Balancing sensitivity and specificity is incredibly difficult when it comes to medical applications, especially in the case of infectious diseases.

When it comes to computer vision and deep training in medicine, you should always remember that our predictive models can have very real consequences - a missed diagnosis can cost a living.

Again, these results are collected for educational purposes only. This article and the accompanying results are not intended for publication in a scientific journal and do not comply with the TRIPOD rules for the creation of forecast reporting models .

Limitations, improvements and future work



Fig. 4. Currently, artificial intelligence (AI) specialists and deep learning specialists suffer from a lack of quality data for the effective training of COVID-19 automatic detection systems on images ( image source )

One of the main limitations for creating such systems is data. We simply do not have enough (reliable) data to train the COVID-19 detector. Hospitals are crowded with COVID-19 patients, but given their rights and privacy, it is becoming even more difficult to collect quality sets of medical images in a timely manner.

I suppose that in the next 12-18 months the problem will be solved, but for now we are content with what we have. I did my best (considering my current mental and physical condition) to compile a guide for those who are interested in using computer vision and deep learning in the fight against COVID-19, taking into account time and resource limitations. But I must remind you that I am not a medical expert.

A true COVID-19 detector must undergo rigorous testing by trained medical professionals working hand in hand with experienced deep learning practitioners. The method described here, of course, is not suitable for practical use and is intended only for educational purposes.

In addition, we must wonder what this model actually “learns”. As I said inby the Grad-CAM guide last week, it is entirely possible that our model studies patterns that are not related to COVID-19, but simply are variations between two data sets (i.e., between patients with and without COVID-19). To confirm the results, thorough testing and expert assistance is required.

Finally, future (and best) detectors will be multimodal.

Now we only consider images (for example, x-rays). The best COVID-19 automated detectors should use several data sources, not limited to images, including patient vital signs, population density, geographical location, etc. Images alone are usually not enough for this type of application.

I hope this guide serves as a starting point for anyone interested in using computer vision and deep learning to automatically detect COVID-19.

What's next?


I usually end my blog articles with a recommendation from one of my books or courses so you can learn more about using computer vision and deep learning. Out of respect for the seriousness of the coronavirus, I'm not going to do this - now is not the time or the place.

Instead, I will say that we are now in a very terrible period of life. Like the seasons, everything will pass, but for now you need to squat down and prepare for the cold winter - it is likely that the worst is yet to come.

I received several letters from PyImageSearch readers who want to use this downtime to study computer vision and deep learning, rather than go crazy in their homes. This is a useful thought.

Summary


In this guide, you learned how to use Keras and TensorFlow to train the COVID-19 automatic detector on a fluorogram set.

There are no high-quality, peer-reviewed image datasets for COVID-19 (yet), so we had to work with what we had, namely, Joseph Cohen's repository on GitHub :


We used Keras and TensorFlow to train the COVID-19 detector, which demonstrates 90-92% accuracy, 100% sensitivity and 80% specificity on the test set (given our limited data set).

Keep in mind that the COVID-19 detector described is for educational purposes only.

More articles:


All Articles