ADAPTIVE MULTI-SPECTRAL VISION FUSION FOR RELIABLE HAZARD IDENTIFICATION IN CHALLENGING VISIBILITY CONDITIONS

Abstract

The degradation of visibility due to fog, smoke and darkness is one of the premier economic concerns today in the operation of industrial and transportation systems. In addressing this issue for industrial and transportation operations, an important factor is the timely identification of hazards that increases the risk of accidents due to delays. Unfortunately, the vast majority of traditional visual systems that utilize visible light sensors to create scene information during degraded atmospheric conditions have failed to do so with a high degree of accuracy or stability. An opportunity exists to develop a multi-spectral fusion architecture using all available illumination sources (visible light, infrared, and thermal) to improve hazard detection. The methodology developed in this paper enables the use of a hybrid form of deep learning through the implementation of a multi-scale convolutional encoder, which generates data-driven decisions for the efficient identification of hazards using an anomaly-detection module to combine feature-level and decision-level fused data. The anomaly-detection module in this method uses a new multi-scale convolutional encoder (multi-scale CE) that enables the presentation of the fused image feature data to the anomaly-detection module, as well as the training of the multi-scale CE using a custom data set that includes an array of images with different opacities and difficulty levels. The multi-scale CE shows improvement in the performance of hazard detection compared to the results of hazard detection from the individual data modalities; for instance, it demonstrated a maximum increase of 32 percent in identification accuracy, a maximum increase of 27 percent in recognition precision, and a relative decrease in the number of non-confirmed negative alerts. The multi-scale CE reliably detected the presence of obstacles and persons at near-zero visibility conditions, and produced nearly identical results across a variety of weather or lighting conditions. In summary, the use of multi-scale CE and the multi-spectral data fusion methods will enable the expansion of the ability to use multi-spectral fused image analysis, thus providing further enhancement of safety and reliability in the identification of hazards and reducing the probability of accidents due to hazards in the most difficult of environmental conditions.