In many sensor applications there is a network of sensors observing the same phenomenon. The sensors may completely duplicate each other, or they may overlap only partially, perhaps observing different angles or aspects of the situation. They may be identical sensors or they could be sensors of entirely different types. Multiple different sensor types may be required as they may be more effective at different speeds, distances or light levels, all of which may vary throughout the observation period. The addition of multiple sensors can create a more accurate and complete picture, but their output also needs to be integrated somehow and reconciled where they disagree.
When creating algorithms to process sensor data from a network of multiple sensors, some complex mathematics can be required to “fuse” the readings together into a single coherent picture. Since modern sensing systems can generate vast amounts of data, efficient fusion algorithms are a necessity to quickly combine and condense the data into manageable quantities.
When sensors are being used to influence decision making of an automated system, deciding which sensors should be given priority and how discrepancies should be resolved can be a matter of critical importance. When the machine controls a manufacturing production line or an aeroplane’s angle, a sub-optimal algorithm that fails to manage it’s many inputs effectively could mean lost money or lost lives.
Sensor fusion can be “competitive”, where sensors present potentially differing measurements of the same quantity, or “cooperative”, where the sensors work together to build up a complete picture.
Sensor fusion for identical sensors making identical measurements
Let’s look first at the simplest case of sensor fusion, which occurs where multiple identical sensors have made the same measurement (this situation could also result from the same sensor making a measurement multiple times). There are two reasons why such a setup is desirable. Firstly, since sensors always have some finite accuracy limitation, multiple measurements are likely to differ due to an element of random noise. By taking multiple measurements, you can average over them to average out some of the noise. This may be a preferable solution to developing or purchasing more accurate (and more expensive) sensors. The second reason for fusing together multiple identical measurements is redundancy to handle sensor errors or sensor failure. When an aeroplane is making decisions based on the readings from a number of sensors, it is critical that it be able to identify and discard the readings from an erroneous sensor.
The simplest way to combine the output from the sensors in this case is simply to take the average of their measurements. Going a little further, if some of the sensors are known to be more accurate than others (measured by the variance in their outputs), you can give more weight to the more accurate sensors in a mathematically precise way. Going to ever further levels of complexity, one can use Kalman filters or Bayesian statistics to combine these measurements.
More problematic is that the practical realities of sensors mean that their readings don’t just suffer from statistical noise, but they can sometimes fail in ways that cause them to give completely random readings, or no reading at all. Thus, it is wise to implement algorithms that can recognise and discard outliers before inputting the data into your sensor fusion algorithm. Fault tolerance is a main motivation of multi-sensor systems in applications that require robustness.
For some applications, it may be known that the data ought to obey some functional relationship, like a straight line or exponential. In this case the averaging is effectively carried out by assuming this functional form and doing a function fitting (i.e. regression). In this case, erroneous readings and sensor errors can be eliminated using algorithms like RANSAC. Furthermore, principal component analysis can be used to discard irrelevant measurements and simplify data.
Sensor fusion for disparate sensor types or sensors making different measurements
Multiple sensors of the same type may be set up in a network to observe different angles or aspects of a situation. A simple example of this would be sensors each observing a different part of the scene, with possibly some degree of overlap. Where the sensor domains overlap, their measurements might be fused using the techniques mentioned in the previous section.
A more interesting situation occurs when sensors are set up in a way that means their readings are not directly comparable. For example, a network of sensors might be observing a scene from different angles. In this case the sensors will sometimes compete with each other, and sometimes cooperate to provide additional information that other sensors can’t see. An effective algorithm must decide which is appropriate at any given time.
Mathematical transformations on sensor data
When sensors are making measurements that are not directly equivalent (such as different angles of the same scene), their outputs cannot be immediately integrated. A similar situation occurs with sensors of different types that measure different physical properties. The sensor outputs are not estimates of the same quantities, but may imply estimates of the same quantities after mathematical calculations are performed on them. In this situation, mathematical transformations must be performed to extract the desired quantities before fusion can occur. Furthermore, their uncertainties cannot be directly compared either. Rather, the sensor uncertainties must be processed through the same calculations to determine the error in the quantities of interest. This leads to some interesting statistics when trying to fuse together the outputs of a diverse sensing system.
Sensor fusion consulting services
Muliple sensor systems are how machines observe the world. As automated devices like self-driving cars become more common, sophisticated sensing systems and the algorithms that drive them will become more mainstream.
We design sensor fusion algorithms for scientists and engineers. We also offer many other sensor data analysis consulting services. See our main article here.