|
|
|
|
|
|
Research on Spectral Reconstruction Method From RGB Imaging Based on Dual Attention Mechanism |
SUN Bang-yong1, YU Meng-ying1, YAO Qi2* |
1. Faculty of Printing, Packaging Engineering and Digital Media Technology, Xi'an University of Technology, Xi'an 710054,China
2. Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
|
|
|
Abstract Spectral images, which theoretically have a wider range of applications, store more information than RGB images. However, due to the high cost of spectral imaging equipment and complex data processing, spectral images are mainly applied in remote sensing, military and other fields. In recent years, scholars have proposed solutions to reconstruct spectral images by mathematical methods using RGB images, which can greatly improve the application range of spectral images. However, there are many problems in current spectral reconstruction models, such as the loss of image details and insufficient spectral accuracy. Therefore, this paper proposes a spectral reconstruction method from RGB images based on a dual attention mechanism to improve the quality of spectral image reconstruction from image detail and spectral accuracy. The proposed spectral reconstruction method designs a sparse signal depth reconstruction network, focusing on the sparse characteristics of RGB images, and achieves sparse to complete signals reconstruction by accurately extracting multi-level features of image information and mining more semantic information. Regarding network structure, the designed spectral reconstruction network first uses small parameter convolution to extract shallow feature information of RGB images. Then, the effective multi-frequency channel attention mechanism was used to calculate the correlation between each channel in the feature layer, and the effective distribution of feature response was realized by inter-layer weighting. At the same time, the layer feature weighted fusion attention mechanism is introduced to learn the dependence between features of different layers, and the weights are optimized through different layers' weighting to extract effective spectral depth features. Finally, based on the extracted depth features, the hyperspectral image is transformed into a specified dimension by convolution. The experiment uses the python 3.7 programming language, pytorch 1.2, as the deep learning model framework and combined spectral image error and RGB image error as loss functions for the training of the spectral reconstruction network. The proposed method and 7 mainstream spectral reconstruction methods are compared and verified on the NTIRE 2020 and CAVE datasets. From a subjective perspective, the spectral image details recovered by this method are clearer, and the error is smaller. From the perspective of objective indicators, the spectral images reconstructed by this method are reduced by 18.9%, 16.6%, and 22.2% in RRMSE, RSAM and RERGAS indicators, respectively, compared with the methods with better reconstruction performance in the existing literature. The RPSNR indicator improved by 4.5%. Therefore, the experimental results prove the effectiveness of the proposed method from RGB image spectral reconstruction.
|
Received: 2022-03-28
Accepted: 2022-10-12
|
|
Corresponding Authors:
YAO Qi
E-mail: yaoqi@fudan.edu.cn
|
|
[1] Parmar M, Lansel S, Wandell B A. IEEE International Conference on Image Processing, 2008: 473.
[2] Arad B, Ben-Shahar O. European Conference on Computer Vision. Amsterdam, Netherlands, 2016: 19.
[3] Aeschbacher J, Wu J, Timofte R. IEEE International Conference on Computer Vision, 2017: 471.
[4] LI Fu-hao, LI Chang-jun(李富豪, 李长军). Spectroscopy and Spectral Analysis(光谱学与光谱分析), 2021, 41(10): 3281.
[5] Xiao G S, Wan X X, Wang L X, et al. Optics Express, 2019, 27(24): 34921.
[6] Liang J X, Wan X X. Optics Express, 2017, 25(23): 28273.
[7] LIANG Jin-xing, WAN Xiao-xia(梁金星,万晓霞). Acta Optica Sinica(光学学报), 2017, 37(9): 0933001.
[8] Jia Y, Zheng Y Q, Gu L, et al. IEEE International Conference on Computer Vision, 2017: 4715.
[9] Liang J X, Xiao K D, Pointer M R. Optics Express, 2019, 27(4): 5165.
[10] Yan Y Q, Zhang L, Li J, et al. Pattern Recognition and Computer Vision. Guangzhou, China, 2018, 11257: 206.
[11] LI Yong, JIN Qiu-yu, ZHAO Huai-ci, et al(李 勇,金秋雨,赵怀慈, 等). Acta Optica Sinica(光学学报), 2021, 41(7): 0730001.
[12] Xiong Z W, Shi Z, Li H Q, et al. IEEE International Conference on Computer Vision Workshop, 2017: 518.
[13] Nathan D S, Uma K, Vinothini D S, et al. arXiv: 2020, 2004.06930v2.
[14] Zhao Y Z, Po L M, Yan Q, et al. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2020: 1695.
[15] LIU Peng-fei, ZHAO Huai-ci, LI Pei-xuan(刘鹏飞,赵怀慈,李培玄). Infrared and Laser Engineering(红外与激光工程), 2020, 49(S1): 20200093.
[16] Li J J, Wu C X, Song R, et al. IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2020: 1894.
[17] Qin Z Q, Zhang P Y, Wu F, et al. IEEE International Conference on Computer Vision, 2021: 763.
[18] Galliani S, Lanaras C, Marmanis D, et al. arXiv: 2017, 1703.09470v1.
|
[1] |
LIU Zhen1*, LIU Li2*, FAN Shuo2, ZHAO An-ran2, LIU Si-lu2. Training Sample Selection for Spectral Reconstruction Based on Improved K-Means Clustering[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2024, 44(01): 29-35. |
[2] |
CHU Bing-quan1, 2, LI Cheng-feng1, DING Li3, GUO Zheng-yan1, WANG Shi-yu1, SUN Wei-jie1, JIN Wei-yi1, HE Yong2*. Nondestructive and Rapid Determination of Carbohydrate and Protein in T. obliquus Based on Hyperspectral Imaging Technology[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(12): 3732-3741. |
[3] |
LIANG Jin-xing1, 2, 3, XIN Lei1, CHENG Jing-yao1, ZHOU Jing1, LUO Hang1, 3*. Adaptive Weighted Spectral Reconstruction Method Against
Exposure Variation[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(11): 3330-3338. |
[4] |
YUAN Wei-dong1, 2, JU Hao2, JIANG Hong-zhe1, 2, LI Xing-peng2, ZHOU Hong-ping1, 2*, SUN Meng-meng1, 2. Classification of Different Maturity Stages of Camellia Oleifera Fruit
Using Hyperspectral Imaging Technique[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(11): 3419-3426. |
[5] |
SHEN Ying, WU Pan, HUANG Feng*, GUO Cui-xia. Identification of Species and Concentration Measurement of Microalgae Based on Hyperspectral Imaging[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(11): 3629-3636. |
[6] |
YANG Lei1, 2, 3, ZHOU Jin-song1, 2, 3, JING Juan-juan1, 2, 3, NIE Bo-yang1, 3*. Non-Uniformity Correction Method for Splicing Hyperspectral Imager Based on Overlapping Field of View[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(11): 3582-3590. |
[7] |
DONG Jian-jiang1, TIAN Ye1, ZHANG Jian-xing2, LUAN Zhen-dong2*, DU Zeng-feng2*. Research on the Classification Method of Benthic Fauna Based on
Hyperspectral Data and Random Forest Algorithm[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(10): 3015-3022. |
[8] |
WANG Yu-chen1, 2, KONG Ling-qin1, 2, 3*, ZHAO Yue-jin1, 2, 3, DONG Li-quan1, 2, 3*, LIU Ming1, 2, 3, HUI Mei1, 2. Hyperspectral Reconstruction From RGB Images for Tissue Oxygen
Saturation Assessment[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(10): 3193-3201. |
[9] |
WEI Zi-kai, WANG Jie, ZHANG Ruo-yu, ZHANG Meng-yun*. Classification of Foreign Matter in Cotton Using Line Scan Hyperspectral Transmittance Imaging[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(10): 3230-3238. |
[10] |
DENG Yun1, 2, NIU Zhao-wen1, 2, FENG Qi-yao1, 2, WANG Yu1, 2*. A Novel Hyperspectral Prediction Model of Organic Matter in Red Soil Based on Improved Temporal Convolutional Network[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(09): 2942-2951. |
[11] |
WU Kuang, SUN Chun, CAO Guan-long*, QIU Bo*, YAO Lin, ZHANG Ming-ru, ZHANG Li-wen. An Algorithm for Redshift Estimation of Photometric Images Using
Convolutional Neural Networks[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(08): 2529-2535. |
[12] |
TANG Ting, PAN Xin*, LUO Xiao-ling, GAO Xiao-jing. Fusion of ConvLSTM and Multi-Attention Mechanism Network for
Hyperspectral Image Classification[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(08): 2608-2616. |
[13] |
MAO Yi-lin1, LI He1, WANG Yu1, FAN Kai1, SUN Li-tao2, WANG Hui3, SONG Da-peng3, SHEN Jia-zhi2*, DING Zhao-tang1, 2*. Quantitative Judgment of Freezing Injury of Tea Leaves Based on Hyperspectral Imaging[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(07): 2266-2271. |
[14] |
LIU Gang1, LÜ Jia-ming1, NIU Wen-xing1, LI Qi-feng2, ZHANG Ying-hu2, YANG Yun-peng2, MA Xiang-yun2*. Detection of Sulfur Content in Vessel Fuel Based on Hyperspectral
Imaging Technology[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(06): 1697-1702. |
[15] |
LI Bin, HAN Zhao-yang, WANG Qiu, SUN Zhao-xiang, LIU Yan-de*. Research on Bruise Level Detection of Loquat Based on Hyperspectral
Imaging Technology[J]. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2023, 43(06): 1792-1799. |
|
|
|
|