Study on the Selection of Spectral Preprocessing Methods
DIWU Peng-yao, BIAN Xi-hui*, WANG Zi-fang, LIU Wei
State Key Laboratory of Separation Membranes and Membrane Processes, School of Environmental and Chemical Engineering, Tianjin Polytechnic University, Tianjin 300387, China
Abstract：Spectral signals of complex samples are usually disturbed by stray light, noise, baseline drift and other undesirable factors, which can affect the final qualitative and quantitative analysis results. Therefore, it is necessary to pretreat the raw spectra before modeling. How to find a proper preprocessing method from the existing spectral preprocessing methods is a difficult problem. One strategy is to choose the optimal preprocessing by observing the characteristics of the spectral signal directly, which does not require modeling and is more explanatory. However, it may be difficult and subjective for subtle or multiple interferences and lead to misleading results. Another strategy is based on the modeling performance, which does not need observe the spectral characteristics, but numerous processing methods need to investigate which is time-consuming for large datasets. In summary, it is necessary to explore which selection method is more scientific and reasonable. In this study, nine datasets were used to investigate the necessity of preprocessing and the choice of preprocessing methods by arranging and combining of 10 preprocessing methods. Firstly, the latent variables of partial least squares (PLS), the window size of first derivative (1st Der), second derivative (2nd Der) and SG smoothing, the wavelet function and decomposition scale of continuous wavelet transform (CWT) were optimized, respectively. Then, non-preprocessing and 10 preprocessing methods including 1st Der, 2nd Der, CWT, multiplicative scatter correction (MSC), standard normal variate (SNV), SG smoothing, mean centering, normalization, Pareto scaling, auto scaling were combined in order of baseline correction, scattering correction, smoothing and scaling. A total of 120 preprocessing and their combinations were obtained. Finally, the characteristics of spectral signals and the root mean squared error of prediction (RMSEP) with PLS for 120 preprocessing methods were analyzed for the nine datasets and the same dataset with different components. Results show that compared with observing the characteristics of spectral signals, the optimal preprocessing method can be selected more accurately according to the modeling performance of the spectra and predictive components. For most datasets, appropriate preprocessing method can improve the modeling performance. For different datasets, the optimal preprocessing method is different because of the different information and complexity of the datasets. For the same dataset, the optimal preprocessing methods for different components are also different even if the spectra are the same. Thus, it can be concluded that no universal preprocessing method exists. The optimal preprocessing method is related to the spectra and the predictive components. Furthermore, it is an effective way to select the optimal pretreatment method by sorting and combining the existing preprocessing methods according to the preprocessing purpose.
第五鹏瑶，卞希慧，王姿方，刘 巍. 光谱预处理方法选择研究[J]. 光谱学与光谱分析, 2019, 39(09): 2800-2806.
DIWU Peng-yao, BIAN Xi-hui, WANG Zi-fang, LIU Wei. Study on the Selection of Spectral Preprocessing Methods. SPECTROSCOPY AND SPECTRAL ANALYSIS, 2019, 39(09): 2800-2806.
[1] Li P, Du G R, Cai W S, et al. Journal of Pharmaceutical and Biomedical Analysis, 2012, 70: 288.
[2] Devos O, Downey G, Duponchel L. Food Chemistry, 2014, 148: 124.
[3] Liu Y J, Yu Y D, Zhou X G, et al. Chemometrics and Intelligent Laboratory Systems, 2017, 161: 8.
[4] Gerretzen J, Szymanska E, Jansen J J, et al. Analytical Chemistry, 2015, 87(24): 12096.
[5] Engel J, Gerretzen J, Szymanska E, et al. Trends in Analytical Chemistry, 2013, 50: 96.
[6] Zhu Z Q, Yuan H F, Song C F, et al. Sensors and Actuators B: Chemical, 2018, 268: 299.
[7] Qiao X X, Wang C, Feng M C, et al. Spectroscopy Letters, 2017, 50(3): 156.
[8] Li Y K, Shao X G, Cai W S. Talanta, 2007, 72(1): 217.
[9] Wulfert F, Kok W T, Smilde A K. Analytical Chemistry, 1998, 70: 1761.
[10] Gerretzen J, Szymanska E, Bart J, et al. Analytica Chimica Acta, 2016, 938: 44.
[11] Bian X H, Li S J, Lin L G, et al. Analytica Chimica Acta, 2016, 925: 16.
[12] Liu P, Wang J, Li Q, et al. Spectrochimica Acta Part A: Molecular and Biomolecular Spectroscopy, 2018, doi:10.1016.
