Research on Spectral Reconstruction Method From RGB Imaging Based on Dual Attention Mechanism
SUN Bang-yong1, YU Meng-ying1, YAO Qi2*
1. Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an 710054，China
2. Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
Abstract：Spectral images, which theoretically have a wider range of applications, store more information than RGB images. However, due to the high cost of spectral imaging equipment and complex data processing, spectral images are mainly applied in remote sensing, military and other fields. In recent years, scholars have proposed solutions to reconstruct spectral images by mathematical methods using RGB images, which can greatly improve the application range of spectral images. However, there are many problems in current spectral reconstruction models, such as the loss of image details and insufficient spectral accuracy. Therefore, this paper proposes a spectral reconstruction method from RGB images based on a dual attention mechanism to improve the quality of spectral image reconstruction from image detail and spectral accuracy. The proposed spectral reconstruction method designs a sparse signal depth reconstruction network, focusing on the sparse characteristics of RGB images, and achieves sparse to complete signals reconstruction by accurately extracting multi-level features of image information and mining more semantic information. Regarding network structure, the designed spectral reconstruction network first uses small parameter convolution to extract shallow feature information of RGB images. Then, the effective multi-frequency channel attention mechanism was used to calculate the correlation between each channel in the feature layer, and the effective distribution of feature response was realized by inter-layer weighting. At the same time, the layer feature weighted fusion attention mechanism is introduced to learn the dependence between features of different layers, and the weights are optimized through different layers' weighting to extract effective spectral depth features. Finally, based on the extracted depth features, the hyperspectral image is transformed into a specified dimension by convolution. The experiment uses the python 3.7 programming language, pytorch 1.2, as the deep learning model framework and combined spectral image error and RGB image error as loss functions for the training of the spectral reconstruction network. The proposed method and 7 mainstream spectral reconstruction methods are compared and verified on the NTIRE 2020 and CAVE datasets. From a subjective perspective, the spectral image details recovered by this method are clearer, and the error is smaller. From the perspective of objective indicators, the spectral images reconstructed by this method are reduced by 18.9%, 16.6%, and 22.2% in RRMSE, RSAM and RERGAS indicators, respectively, compared with the methods with better reconstruction performance in the existing literature. The RPSNR indicator improved by 4.5%. Therefore, the experimental results prove the effectiveness of the proposed method from RGB image spectral reconstruction.
孙帮勇，喻梦莹，姚 其. 基于双重注意力机制的RGB成像光谱重建方法研究[J]. 光谱学与光谱分析, 2023, 43(09): 2687-2693.
SUN Bang-yong, YU Meng-ying, YAO Qi. Research on Spectral Reconstruction Method From RGB Imaging Based on Dual Attention Mechanism. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(09): 2687-2693.
[1] Parmar M, Lansel S, Wandell B A. IEEE International Conference on Image Processing, 2008: 473.
[2] Arad B, Ben-Shahar O. European Conference on Computer Vision. Amsterdam, Netherlands, 2016: 19.
[3] Aeschbacher J, Wu J, Timofte R. IEEE International Conference on Computer Vision, 2017: 471.
[4] LI Fu-hao, LI Chang-jun（李富豪, 李长军）. Spectroscopy and Spectral Analysis(光谱学与光谱分析), 2021, 41(10): 3281.
[5] Xiao G S, Wan X X, Wang L X, et al. Optics Express, 2019, 27(24): 34921.
[6] Liang J X, Wan X X. Optics Express, 2017, 25(23): 28273.
[7] LIANG Jin-xing, WAN Xiao-xia（梁金星，万晓霞）. Acta Optica Sinica(光学学报), 2017, 37(9): 0933001.
[8] Jia Y, Zheng Y Q, Gu L, et al. IEEE International Conference on Computer Vision, 2017: 4715.
[9] Liang J X, Xiao K D, Pointer M R. Optics Express, 2019, 27(4): 5165.
[10] Yan Y Q, Zhang L, Li J, et al. Pattern Recognition and Computer Vision. Guangzhou, China, 2018, 11257: 206.
[11] LI Yong, JIN Qiu-yu, ZHAO Huai-ci, et al（李 勇，金秋雨，赵怀慈, 等）. Acta Optica Sinica(光学学报), 2021, 41(7): 0730001.
[12] Xiong Z W, Shi Z, Li H Q, et al. IEEE International Conference on Computer Vision Workshop, 2017: 518.
[13] Nathan D S, Uma K, Vinothini D S, et al. arXiv: 2020, 2004.06930v2.
[14] Zhao Y Z, Po L M, Yan Q, et al. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2020: 1695.
[15] LIU Peng-fei, ZHAO Huai-ci, LI Pei-xuan（刘鹏飞，赵怀慈，李培玄）. Infrared and Laser Engineering(红外与激光工程), 2020, 49(S1): 20200093.
[16] Li J J, Wu C X, Song R, et al. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2020: 1894.
[17] Qin Z Q, Zhang P Y, Wu F, et al. IEEE International Conference on Computer Vision, 2021: 763.
[18] Galliani S, Lanaras C, Marmanis D, et al. arXiv: 2017, 1703.09470v1.
