Autonomous vehicles and mobile robots are a fast-growing area of engineering research. They always raise the same question: how can a car drive itself in real traffic, or how does a robot see its environment well enough to do its job? Let’s DIVE into the main players behind these machines.
Sensors
Sensors are the technology behind autonomous vehicles and systems: they act as the eyes of vehicles and mobile robots. A great deal of research and experimentation has gone into making them robust. OEMs and other providers use different types of sensors, but the most widely adopted are vision sensors (cameras), LiDARs (Light Detection and Ranging), radars, GPS (Global Positioning System) and GNSS (Global Navigation Satellite System).
Visual sensors
Cameras are the most widely used sensors for building machine vision and perception for mobile robots and autonomous vehicles. There are several types: stereo cameras, RGB-D cameras (with an additional depth sensor for a more accurate depth image of the environment), fish-eye cameras for a wider field of view, and more. Vision sensors are cost-effective compared with other perception sensors, but they can fail in many situations: entering dark areas, bad weather (rain, fog, snow), missing lane markings and so on.
Credit: A survey on vision-based UAV navigation.
LiDAR: Light Detection and Ranging
Credit: Articulated Robotics.
LiDAR technology was adapted from the aviation industry, where it plays a crucial role in accurately estimating an aircraft’s position and distance. In autonomous systems, LiDAR opened a new area of development, and a great deal of research is going into improving it. LiDAR sensors have been widely adopted in the automotive industry, by Volvo (a major player in safe autonomous platforms), Waymo (Google’s self-driving company), Zoox and many others. But how does it work?
LiDAR works in three stages. First, a laser beam, or a set of hundreds of beams, is sent out from the sensor, which rotates 360° to spread the beams all around and gather data. Second, when the beams hit an object or obstacle (vehicles, pedestrians, cyclists, poles, road signs), they are reflected back and picked up by the receiver. Finally, the distance to the object is calculated from the time the beam takes to travel out and back, known as the time of flight. LiDAR also has the advantage of producing a point cloud for estimation tasks. Still, there are scenarios where LiDAR struggles to detect accurately. To overcome the limits of both vision sensors and LiDAR, the sensors are fused together.
4D radar
A general representation of a 4D radar detecting pedestrians, other vehicles and objects.
4D radar sensors are still being researched for use in autonomous vehicles. Unlike cameras and LiDAR, 4D imaging radar uses echolocation and time-of-flight measurement to locate objects. It works in all weather and environments, including fog, torrential rain and night-time. Even in a complete white-out, it can detect where a car is and how fast it is moving. It can also sense out to 300 metres, which higher levels of automation need.
The concept
A 4D radar measures the distance, relative speed and azimuth of surrounding objects, plus their elevation (height). The fourth dimension is often described as time; these radars use it to understand the 3D environment in terms of elevation rather than actually mapping time. This helps a vehicle tell whether a stationary object ahead is a person or a tree branch. Unlike conventional radars, the sensor is packed with multiple-input multiple-output (MIMO) antennas that transmit signals to objects in the environment while receivers collect the reflected signals. The data is used to build a point-cloud map of the surroundings for detection and tracking algorithms. Another advantage is that, because it doesn’t rely on cameras, a 4D radar can sense objects that are hidden behind others.
Sensor fusion
Sensor fusion is the approach researchers and companies use to combine sensors for robust, real-time results. Data from many sensors is collected while the vehicle moves and processed by algorithms for object detection and tracking, localisation and mapping, path planning and control. Data from sensors on the vehicle or robot, such as stereo cameras, IMUs (inertial measurement units, which provide odometry: the vehicle’s current position relative to its start) and LiDARs, is passed through a Kalman filter. For autonomous vehicles (Ackermann-steered) and robots (differential-drive, 4WD), the Extended Kalman Filter (EKF) is the most commonly used algorithm for fusing multi-sensor information.
Cameras give the best images during the day, while LiDAR collects point clouds of the environment. These are used for localisation (where the vehicle is in its environment) and mapping (a map of the surroundings with the obstacles detected), achieved with state-of-the-art simultaneous localisation and mapping (SLAM) algorithms.
Conclusion
This post covers the main sensors (cameras, LiDARs and radars) and gives a brief introduction to sensor fusion.