A Spike Removal Algorithm Based on Median Filter and Statistic for
Raman Spectra
YE Rui-qian1, HE Hao1, ZHENG Peng1, XU Meng-xi2, WANG Lei1*
1. School of Aerospace Engineering, Xiamen University, Xiamen 361101, China
2. State Key Laboratory of Physical Chemistry of Solid Surfaces, Collaborative Innovation Center of Chemistry for Energy Materials (iChEM), College of Chemistry and Chemical Engineering, Xiamen University, Xiamen 361101, China
Abstract：Raman spectroscopy is a promising technique widely used in chemistry, biology, and physics. However, as the key part of the Raman spectrometer, the charge-coupled device is vulnerable to cosmic rays, resulting in a random narrow bandwidth and a high-intensity spikes. It will cause a significant reduction in signal contrast. In this paper, we propose a practical spike removal algorithm. Firstly, the algorithm obtains deviation data by separating the median filtered data from the original data. Then, deviation data is sorted from small to large by quantile method, and the intermediate 99% data are selected for Gaussian distribution fitting. Considering the characteristics of high-intensity and sparsity of the spike, the occurrence probability of high intensity data in the spectra is used as the threshold standard to remove spike. Finally, the spikes are replaced by new data using median filtered at corresponding positions. This algorithm restores the original sample information without any debugging parameters. Different intensities of spikes are added in Raman spectra to verify the algorithm, and the experimental results show that this algorithm’s sensitivity can reach 99.5%. Besides, this algorithm is applicable for one-dimensional Raman spectra, two-dimensional Raman images and three-dimensional Raman data cubes, and the performance improves with the increase of dimensionality. Specifically, the one-dimensional spike removal algorithm can detect spikes exceeding 40% of the maximum peak intensity. The Raman data cubes can be detected exceeding 20% of the peak value. The algorithm is used to process 40 000 real Raman spectra and can effectively remove spikes without distorting the real signal, proving the algorithm’s practicability.
Key words：Raman spectroscopy; Cosmic rays; Median filter; Spike removal
叶瑞乾，何 浩，郑 鹏，徐梦溪，王 磊. 一种基于中值滤波和统计学的拉曼光谱spike剔除算法[J]. 光谱学与光谱分析, 2022, 42(10): 3174-3179.
YE Rui-qian, HE Hao, ZHENG Peng, XU Meng-xi, WANG Lei. A Spike Removal Algorithm Based on Median Filter and Statistic for
Raman Spectra. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2022, 42(10): 3174-3179.
[1] YAN Jing-wen, LI Ying, CHEN Liang, et al（闫静文, 李 颖, 陈 靓, 等）. Journal of Atmospheric and Environmental Optics(大气与环境光学学报). 2017, 12(2): 128.
[2] REN Bin(任 斌). Optics & Optoelectronic Technology(光学与光电技术)，2021, 19(4): 1.
[3] LIANG Ying-fang, ZHOU Hua-lan, WANG Yan, et al(梁营芳, 周化岚, 王 燕, 等). Physical Testing and Chemical Analysis Part B：Chemical Analysis(理化检验：化学分册)，2020, 56(4): 487.
[4] TIAN Hui-yan, LIU Yu, HUANG Jiao-qi, et al(田晖艳, 刘 羽, 黄姣祺, 等). Spectroscopy and Spectral Analysis(光谱学与光谱分析)，2020, 40(10): 3021.
[5] LIU Xia, HUO Ya-peng, KANG Wei-jun, et al(刘 厦, 霍亚鹏, 康维钧, 等). Chinese Science Bulletin(科学通报)，2020, 65(15): 1448.
[6] Schulze H G, Turner R F. Applied Spectroscopy，2013，67(4): 457.
[7] Tian Y, Burch K S. Applied Spectroscopy，2016，70(11): 1861.
[8] Zhao J. Applied Spectroscopy，2003，57(11): 1368.
[9] Huang Z, Zeng H, Hamzavi I, et al. Optics Letters，2001，26(22): 1782.
[10] Lee J S, Cox D D. Computational Statistics & Data Analysis，2010，54(12): 3131.
[11] Katsumoto Y, Ozaki Y. Applied Spectroscopy，2003，57(3): 317.
[12] Silveira Jr L, Bodanese B, Zangaro R A, et al. Instrumentation Science Technology，2010，38(4): 268.
[13] Veneri G, Federighi P, Rosini F, et al. Journal of Neuroscience Methods，2011，196(2): 318.
[14] Zhang D, Ben-Amotz D. Applied Spectroscopy，2002，56(1): 91.
[15] Takeuchi H, Hashimoto S, Harada I. Applied Spectroscopy，1993，47(1): 129.
[16] Uckert K, Bhartia R, Michel J. Applied Spectroscopy，2019，73(9): 1019.
[17] Sharan T S, Sharma S, Sharma N. Applied Spectroscopy，2021，88(1): 117.
[18] LI Sheng, DAI Lian-kui（李 晟, 戴连奎）. The Journal of Light Scattering（光散射学报），2011, 23(3): 188.
[19] Ryabchykov O, Bocklitz T, Ramoji A, et al. Chemomtrics and Intelligent Laboratory Systems，2016，155: 1.
