LiDAR is a sensor currently changing the world. It is integrated in self-driving vehicles, autonomous drones, robots, satellites, rockets, and many more.
This sensor understands the world in 3D using laser beams and measuring the time it takes for the laser to come back. The output is a point cloud.
It’s a technology harder to understand than the camera because less people have access to it. I had access to it and today will help you understand how LiDAR detection works for self-driving cars.
LiDAR’s underlying technology could power a wide array of use cases, but currently, self-driving vehicles offer us the best method of exploring this technology.
Today, I’ll provide an introduction to LiDAR and explain how it works from the self-driving car perspective. I will then explore how to process the point cloud, detect obstacles using 3D bounding boxes, and segment the drivable area in real-time.
The target goal is something similar to this picture.
Before Anything
- Subscribe to the daily emails_ and learn cutting-edge Computer Vision & Self-Driving Cars every day!_
- Visit thinkautonomous.ai and get at the leading-edge of Autonomous Technologies
LiDAR — A 3D Light Sensor
A LiDAR sensor works with light. LiDAR stands for Light Detection And Ranging. They can detect obstacles up to 300 meters and accurately estimate their positions. In Self-Driving Cars, this is the most accurate sensor for position estimation.
64–32–16 Layers LiDARs
A LiDAR sensor is composed of two parts: laser emission (top) and **laser reception **(bottom). The emission system works by leveraging layers of laser beams. The more layers, the more accurate the LiDAR will be. On the image above, you can see that the more layers, the bigger the sensor.
Lasers are sent to obstacles and reflect
When these lasers hit an obstacle, they create a set of point clouds. The sensor works with Time of Flight (TOF). Essentially, it measures the time it takes for every laser beam to reflect and come back.
(Image taken from my self-driving car course)
When at very high quality (and price), LiDARs can create rich 3D scenes of an environment. They can emit up to 2 million points every second.
Point clouds represent the world in 3D. The LiDAR sensor gets the exact (X,Y,Z) position of every impact point.
A LiDAR sensor can be solid-state or rotating
In the first case, it will focus its detection on a position and offer a coverage range (90° for example). In the latter case, it will rotate around itself and offer a 360° view. In this case, we want to place it on the roof for better visibility.
**LiDARs are rarely used as standalone sensors. They’re often coupled with a camera or a RADAR, in a process called Sensor Fusion. **Please find my Sensor Fusion article here to learn more.
The fusion process can be of two types: Early Fusion or **Late Fusion. **In the first case, the point cloud is fused with the image pixels. In the second case, the individual detections are fused together.
What are the disadvantages of LiDAR sensors?
- LiDARs cannot directly estimate velocities. They need to compute the difference between two consecutive measurements to do so. RADARs, on the other hand, can estimate the velocity thanks to the Doppler effect.
- LiDARs don’t work well in bad weather conditions. In cases of fog, the lasers can hit it and muddle the scene. Similar with rain drops or dirt.
- LiDARs are cumbersome in terms of size—they can’t be hidden like a camera or a RADAR.
- The price of LiDAR, even though dropping, is still very high.
What are the advantages of LiDARs?
- LiDARs can accurately estimate the position of obstacles. So far, we don’t have more accurate means to do this.
- LiDARs work with point clouds. If we see point clouds in front of a vehicle, we can stop even if the obstacle detection system didn’t detect anything. It is a huge security for any customer to know that the vehicle won’t only rely on neural networks and probabilities.
In this article, I will cover the obstacle detection process (late fusion) that generally happens in 4 steps:
- Point cloud processing
- Point cloud Segmentation
- Obstacle clustering
- Bounding box fitting.
The future of machine learning is on the edge. Subscribe to the Fritz AI Newsletter to discover the possibilities and benefits of embedding ML models inside mobile apps.
Point Cloud Processing — Voxel Grid
To process point clouds, we can use the most popular library, called PCL (Point Cloud Library). It’s available in Python, but it makes more sense to use it in C++, as the language is more suited to robotics. It’s also compliant with ROS (Robotic OS).
The PCL library can do most of the computations necessary to detect obstacles, from loading the points to executing algorithms. This library is the computer vision equivalent to OpenCV.
Since the output of LiDAR can easily be 100,000 points per second, we need to use something called a voxel grid to downsample the results.
#machine-learning #lidar #tech #heartbeat #self-driving-cars #deep learning
