Semantic segmentation is the problem of detecting and delineating each object of interest appearing in an image. Currently, there are several approaches that solve this problem and produce results as seen below.

Figure 1: Semantic segmentation example (Source)

This kind of segmentation is predicting every pixel in the image and is known as Dense Prediction as well. It’s important to notice that the instances of the same class are not being separated, the model only cares about the pixel’s category. As shown in Figure 1, the method can say that there are chairs in certain positions, but cannot distinguish them.

One of the main applications of this technique is in Autonomous Vehicles, where cars need to understand their environment. Semantic Segmentation is able to assign a meaning to the scenes and put the car in the context, indicating the lane position, if there is some obstruction, as fallen trees or pedestrians crossing the road, and recognizing other cars.

Video 1: Example of Semantic Segmentation for Autonomous Driving

Therefore, applying Semantic Segmentation algorithms in urban street scenes is one of the main Computer Vision challenges nowadays. A popular dataset to evaluate model performance is the Cityscapes. It contains 30 classes from 50 different cities varying the season and wheater conditions.

Figure 2 shows how different algorithms have reached the state of the art in this dataset over time.

Figure 2: Semantic Segmentation on Cityscapes dataset

In this project, it was used a mid-level model that can deliver a reasonable precision and run in real-time. The RefineNet [2] was firstly introduced at the end of 2016 from researches of The University of Adelaide and converted to a Light-Weight model in 2018 [3], allowing real-time inferences.

#tensorflowjs #machine-learning #artificial-intelligence #tensorflow