In computer vision, homography is a transformation matrix H when applied on a projective plane maps it to another plane (or image). In the case of Inverse Perspective Mapping (IPM), we want to produce a birds-eye view image of the scene from the front-facing image plane.
In the field of autonomous driving, IPM aids in several downstream tasks such as lane marking detection, path planning and intersection prediction solely from using a monocular camera as this orthographic view is scale-invariant. Emphasising the importance of this technique.
IPM first assume the world to be flat on a plane. Then it maps all pixels from a given viewpoint onto this flat plane through homography projection.
In practice, IPM works well in the immediate vicinity of the camera. For faraway features in the scene, blurring and stretching of the scene become more prominent during perspective projection as a smaller number of pixel is represented, limiting the application of IPM. This can be observed in figure 1, where severe undesirable distortion is produced farther away. To be exact, the lookahead distance is approximately 50m in the figure.
In addition, the following constraints must hold:
In this article, I will attempt to explain the idea of IPM. More importantly, this post is dedicated to how we can work out and apply the homography using only Python and Numpy.
Next, I will show that somewhat similar results can be obtained using OpenCV. We will be using a relatively common road scene from Cityscape dataset as an example. Feel free to message me regarding any questions/doubts or even mistakes you might have spotted.
The code for this article is available here. And all inline
text refers to some variable or function in the code.
The problem that we are trying to solve is to transform a frontal view image into a birds-eye view image. IPM does this by removing the perspective effect from the front-facing camera and remap its image onto a top-view 2D domain. The image in BEV is one that attempts to preserve distance and parallel lines, correcting perspective effect.
The following bullet points summarise the procedure for homography based IPM:
I will go into more details in the following sections.
Figure 2. Source
Before we get started, it is important to know where the camera is relative to the road. For the road scene in Cityscape, the camera is mounted on top of the vehicle pitching slightly downwards. The exact position and orientation is written in the file camera.json
. Note that the values are relative to the ego-vehicle.
To understand IPM, some background knowledge about perspective projection and camera projective geometry is required. I will briefly describe what you need to know about those topics in this section.
Perspective projection precisely describes how the world around us gets mapped on a 2D plane. During this mapping, 2 parallel lines in the world (euclidean space) are transformed into a pair of line in the new plane which converges at the point of infinity. Referring to figure 3, the parallel properties of the 2 cubes placed in the world are not preserved from where we are observing right now. An example of this would be the lane lines.
Figure 3
In the case of our example as shown in figure 4, when the road is observed from a different viewpoint, notice that the same region looks different. Parallel lies are no longer preserved in a projective transformation P.
#mathematics #machine-learning #computer-vision #deep learning