MASK-RCNN for finding roofs from drone images

Problems

• there is a significant difference between winter and summer aerial photographs (a model trained only in summer photographs is completely unable to find roofs in winter);
• , ;
• , ( ), ( ) , ;
• , , ( ). ;
• (, ) .

• we had no task to perform inference for a limited time (for example, directly at the time of the flight), which immediately solved all possible problems with performance;
• at the input for processing, we immediately received high-quality high-resolution images (using lenses with a focal length of 21 mm at a height of 250 m, which is 5 cm / px) from our customer, the Shakhty company, could use their expertise in the geolocation of objects on maps, and they also had the opportunity to establish a specific set of requirements for future UAV flights, which ultimately greatly reduced the likelihood of very unique tiles that were not in the training set;

The first solution to the problem, stroke using the Boundary box 

• Anaconda is a convenient package management system for Python and R.
• Tensorflow is an open source machine learning software library developed by Google.
• Keras is an add-on for the frameworks Deeplearning4j, TensorFlow and Theano.
• OpenCV is a library of algorithms for computer vision, image processing, and general-purpose open-source numerical algorithms.
• Flask is a framework for creating web applications in the Python programming language.

Tile Preparation

from pyproj import Proj, transform

class Converter:
    P3857 = Proj(init='epsg:3857')
    P4326 = Proj(init='epsg:4326')
...
    def from_3857_to_GPS(self, point):
        x, y = point
        return transform(self.P3857, self.P4326, x, y)
    def from_GPS_to_3857(self, point):
        x, y = point
        return transform(self.P4326, self.P3857, x, y)
...

Training data markup 

• / . :

• , , :

...

lines = cv2.HoughLinesP(edges, 1, np.pi/180, 50, minLineLength=minLineLength, maxLineGap=5)
if lines is not None:
    length = image.shape[0]
    angles = []
    for x1, y1, x2, y2 in lines[0]:
        angle = math.degrees(math.atan2(y2 — y1, x2 - x1))
        angles.append(angle)
    parts_angles.append(angles)
    median_angle = np.median(angles)
...

#     

for x in range(0, image.shape[0]-1, image.shape[0] // count_crops):
    for y in range(0, image.shape[1]-1, image.shape[1] // count_crops):
        get_line(image[x:x+image.shape[0]//count_crops, y:y+image.shape[1]//count_crops, :])
...

#      

np.median([a if a>0 else 90+a for a in np.array(parts_angles).flatten()])

h, w = image.shape[:2]
image_center = (w/2, h/2)

if size is None:
    radians = math.radians(angle)
    sin = math.sin(radians)
    cos = math.cos(radians)
    size = (int((h * abs(sin)) + (w * abs(cos))), int((h * abs(cos)) + (w * abs(sin))))
    rotation_matrix = cv2.getRotationMatrix2D(image_center, angle, 1)
    rotation_matrix[0, 2] += ((size[0] / 2) — image_center[0])
    rotation_matrix[1, 2] += ((size[1] / 2) — image_center[1])
else:
    rotation_matrix = cv2.getRotationMatrix2D(image_center, angle, 1)

cv2.warpAffine(image, rotation_matrix, size)

• there are many overflights where there are a large number of non-rectangular roofs, because of this there is too much manual work to refine the area,
• sometimes found at home with different orientations on the same tile,
• sometimes there are many false lines on the tiles, which ultimately leads to a wrong turn. It looks like this:

The final solution based on Mask-RCNN

• standard for all convolutional neural networks, the allocation of features in the image, such as lines, bends, contrasting boundaries and others;
• Region Proposal Network (RPN) scans small fragments of the image, called anchors (anchors) and determines whether this anchor contains signs characteristic of the target class (in our case, the roof);
• Region of Interest Classification and Bounding Box. At this stage, the network, based on the results of the previous stage, is trying to highlight large rectangular areas in the photograph, presumably containing the target object;
• Segmentation Masks. At this stage, the mask of the desired object is obtained from the rectangular area obtained by applying the boundary box.

Metrics used to evaluate model quality

from shapely.geometry import Polygon

true_polygon = Polygon([(2, 2), (2, 6), (5, 6), (5, 2)])
predicted_polygon = Polygon([(3, 3), (3, 7), (6, 7), (6, 3)])
print(true_polygon.intersection(predicted_polygon).area / true_polygon.union(predicted_polygon).area)

>>> 0.3333333333333333

• val_mrcnn_bbox_loss - shows how well the model locates objects (i.e. imposes a boundary box);
• val_mrcnn_mask_loss - shows how well the model segments objects (i.e. imposes a mask).

Model training and validation

Creating a background service for recognition

sudo vim my_srv.service

• [Unit] - describes the order and condition of the start (for example, you can tell the process to wait for the start of a certain service and only then start it yourself);
• [Service] - describes startup parameters;
• [Install] - describes the behavior of the service when adding it to startup.

[Unit]
Description=my_test_unit

[Service]
WorkingDirectory=/home/user/test_project
User=root
ExecStart=/home/user/test_project/venv/bin/python3 /home/user/test_project/script.py

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl start my_srv.service

findings