1. School of Mechanical Engineering, Key Laboratory of Manufacturing Equipment Mechanism Design and Control of Chongqing, Chongqing Technology and Business University, Chongqing 400067, China
2. Engineering Research Centre for Waste Oil Recovery Technology and Equipment of Ministry of Education, Chongqing Technology and Business University, Chongqing 400067, China
Abstract：To enhance the visibility of infrared and visible image fusion and overcome the problems of detail loss, insignificant target, and low contrast in infrared and visible image fusion results, a novel infrared and visible image fusion method based on two-scale decomposition and saliency extraction is proposed. Firstly, based on the theory of human visual perception, the source image is decomposed at different levels to avoid mixing high-frequency and low-frequency components to reduce the halo effect. In this paper, we use a two-scale decomposition method to decompose the source infrared and visible images and obtain the basic layer and detail layer, respectively, representing the image well and having good real-time performance. Then, a weighted average fusion rule based on a visual saliency map (VSM) is proposed to fuse basic layers, and the VSM method can extract the salient structures and targets in the source images. The VSM-based weighted average fusion rule is used to fuse the base layer, effectively avoiding the contrast loss caused by the direct use of the weighted average strategy and making the fused image perform better. The Kirsch operator is used to extract the source images separately to obtain the salient maps for the fusion of the detail layer. Then the VGG-19 network is applied to get the weight maps by extracting features from the salient maps and fusing them with the detail layer to obtain the fused detail layer. The Kirsch operator can quickly extract the image edges in eight directions, and the significant map will contain more edge information and less noise. The VGG-19 network can extract deeper feature information from the image, and the obtained weight map will have more helpful information. Finally, the fused basic and detail layer images are superimposed to get the final fusion result. Four sets of typical infrared and visible images are selected for testing and compared with six other current mainstream methods in the experimental part. The experimental results show that the method in this paper has the advantages of high contrast, prominent target, rich detail information and better retention of image edge features in terms of subjective quality. The objective metrics such as information entropy, mutual information, standard deviation, multiscale structural similarity measure and difference correlation sum also show relatively good results.
Key words：Infrared and visible image fusion;Two-scale decomposition;Kirsch operator;Weight map;Feature extraction
[1] Wu M H, Ma Y, Huang J, et al. Infrared Physics & Technology, 2020, 110: 103362.
[2] Liu Y, Wang L, Cheng J, et al. Information Fusion, 2020, 64: 71.
[3] Li S T, Kang X D, Fang L Y, et al. Information Fusion, 2017, 33: 100.
[4] Shrinidhi V, Yadav P, Venkateswaran N. IR and Visible Video Fusion for Surveillance. 2018 International Conference on Wireless Communications, Signal Processing and Networking, 2018: 1.
[5] Jiang M X, Deng C, Shan J S, et al. Information Fusion, 2019, 50: 1.
[6] Lewis J J, O’Callaghan R J, Nikolov S G, et al. Information Fusion, 2007, 8(2): 119.
[7] Nencini F, Garzelli A, Baronti S, et al. Information Fusion, 2007, 8(2): 143.
[8] Feng X, Fang C, Lou X C, et al. IEEE Access, 2021, 9: 23498.
[9] SHEN Yu, YUAN Yu-bin, PENG Jing（沈 瑜，苑玉彬，彭 静）. Spectroscopy and Spectral Analysis（光谱学与光谱分析）, 2021, 41(7): 2023.
[10] Bavirisetti D P, Xiao G, Liu G. Multi-Sensor Image Fusion Based on Fourth Order Partial Differential Equations. 2017 20th International Conference on Information Fusion, 2017.
[11] Nirmalraj S, Nagarajan G. ICT Express, 2021, 7(3): 350.
[12] LOU Xi-cheng, FENG Xin(娄熙承，冯 鑫). Acta Photonica Sinica(光子学报)，2021，50(3): 031004.
[13] GAO Cheng-ying, XU Xian-er, LUO Yan-mei, et al（高成英，徐仙儿，罗燕媚，等）. Chinese Journal of Computers（计算机学报）, 2019, 42(9): 1954.
[14] Li H, Wu X J, Kittler J. Infrared and Visible Image Fusion Using a Deep Learning Framework. 2018 24th International Conference on Pattern Recognition, 2018: 2705.
[15] Li H, Wu X J, Durrani T S. Infrared Physics & Technology, 2019, 102: 103039.
[16] Bavirisetti D P, Dhuli R. Infrared Physics & Technology, 2016, 76: 52.
[17] Zhai Y, Shah M. Visual Atlention Detection in Video Sequences Using Spatiotemporal Cues, in Proceedings of the 14th ACM International Conference on Multimedia. 2006: 815.
