Choosing the best model architecture and pretrained weights for your task can be hard. If you’ve ever worked on an object detection problem then you’ve undoubtedly come across plots and tables similar to those below while comparing different models.

Right image source: YOLOv4 [3]. Left image source: EfficientDet [4]

The main thing that you get out of comparisons like these is which model has a higher mAP on the COCO dataset than other models. But how much does that really mean to you? You need to stop strictly looking at aggregate metrics, look instead at the data and model results in more detail to understand what’s working and what’s not.

In recent years, great strides are being made to provide similar detection results with faster models, meaning mAP is not the only factor to consider when comparing two detectors. However, no matter how fast your model is, it still needs to provide high-quality detections that meet your requirements.

While it is important to be able to compare different models easily, reducing the performance of a model down to a single number (mAP) can obscure the intricacies in the model results that may be important to your problem. You should also be considering:

• Bounding box tightness (IoU)
• High confidence false positives
• Individual samples to spot check performance
• Performance on classes most relevant to your task

What is mAP?

Mean average precision (mAP) is used to determine the accuracy of a set of object detections from a model when compared to ground-truth object annotations of a dataset.

We won’t go into full detail here, but you should understand the basics. There is a wide selection of posts discussing mAP in more detail if you are interested [6,_ 7_].

IoU

Intersection over Union (IoU) is used when calculating mAP. It is a number from 0 to 1 that specifies the amount of overlap between the predicted and ground truth bounding box.

• an IoU of 0 means that there is no overlap between the boxes
• an IoU of 1 means that the union of the boxes is the same as their overlap indicating that they are completely overlapping

#object-detection #machine-learning #visualization #evaluation #fiftyone