Adaptive deep prior for hyperspectral image super-resolution
-
摘要: 为了解决现有的高光谱超分辨率方法依赖于手工先验和数据驱动先验会导致参数选择困难或可解释性差的问题，采用一种基于自适应深度先验正则的高光谱图像超分辨率方法，进行了理论分析和实验验证。首先设计基于卷积神经网络的多阶段特征提取网络，提取退化图像的空间和光谱信息；其次将提取到的空-谱先验输入基于transformer模型的特征融合模块；然后自适应交互空域和谱域的互补信息，以捕获图像的全局先验特征；最后在退化模型中插入深度先验正则项，将超分辨率问题表述为一个优化问题，其解可以通过交替方向乘子法获得并降低求解复杂度。结果表明，所提出算法在信噪比均为35 dB时，重建信噪比分别达到了34.16 dB和29.35 dB, 比次优算法高出2.78 dB和2.17 dB，重建的高分辨率高光谱图像与其固有结构具有较高的一致性。该研究为综合利用手工先验和数据驱动先验增强高光谱图像空间分辨率提供了参考。Abstract: To address the issue that the difficult parameter selection or poor interpretability caused by the existing hyperspectral super-resolution methods rely on manual or data-driven prior, a deep prior regularization-based hyperspectral image super-resolution method was adopted for theoretical analysis and experimental validation. First, a multi-stage feature extraction network based on a deep convolutional neural network was designed to extract spatial and spectral information from the degraded images. Then, the collected spatial-spectral prior features were fed into a transformer-based feature fusion module, where complementary information from the spatial and spectral domains was adaptively extracted to capture the image' s global prior features. Finally, the super-resolution problem of the image was formulated as an optimization problem by inserting deep prior regularization term in the degraded model, the solution of which can be achieved using the alternating direction method of multipliers while minimizing solution complexity. Experimental results show that reconstruction signal-to-noise ratio of this algorithm is 34.16 dB and 29.35 dB when both of the signal-to-noise ratio are 35 dB, which is 2.78 dB and 2.17 dB higher than the suboptimal algorithm, respectively. The reconstructed high-resolution hyperspectral images have high consistency with their inherent structures under the condition of deep prior regularization. This study provides a reference for the combined use of manual and data-driven prior to enhance the spatial resolution of hyperspectral images.
-
-
![]()
图 1 基于自适应特征提取的高光谱图像超分辨率方法的流程图
Figure 1. Flowchart of hyperspectral image super-resolution method based on adaptive feature extraction
![]()
图 5 a—深度特征融合模块b—swin-transformer层
Figure 5. a—deep feature fusion module b—swin-transformer layer
![]()
图 6 算法的RMSE性能随正则化参数α的变化曲线
Figure 6. Curve of RMSE performance of the algorithm changes with the variation of the regularization parameter α
![]()
图 7 各算法在CAVE数据集上的重建图像和误差图像
Figure 7. The reconstructed and error maps on CAVE dataset by the different algorithms
![]()
图 8 各算法在Harvard数据集上的重建图像和误差图像
Figure 8. The reconstructed and error maps on Harvard dataset by different algorithms
表 1 CAVE数据集中20%图像测试结果的平均性能
Table 1 Average performance of test results for 20% of the images in the CAVE dataset
algorithm case 1(SNR: 30 dB) case 2(SNR: 35 dB) ERGAS RSNR RMSE SAM SSIM ERGAS RSNR RMSE SAM SSIM SRTR 2.84 25.92 0.10 0.148 0.81 2.84 25.92 0.11 0.148 0.81 CSTF 2.00 28.93 0.08 0.096 0.87 1.52 31.33 0.06 0.065 0.89 BSR 2.72 26.23 0.11 0.121 0.90 2.70 26.31 0.10 0.121 0.90 CNMF 2.03 28.81 0.08 0.084 0.89 1.51 31.38 0.05 0.059 0.93 NLSTF 2.75 26.19 0.10 0.157 0.83 1.65 30.61 0.06 0.097 0.91 DHSR 1.61 30.68 0.06 0.069 0.95 1.05 34.16 0.04 0.046 0.97 
下载: 导出CSV
表 2 Harvard数据集中20%图像测试结果的平均性能
Table 2 Average performance of test results for 20% of the images in the Harvard dataset
algorithm case 1(SNR: 30 dB) case 2(SNR: 35 dB) ERGAS RSNR RMSE SAM SSIM ERGAS RSNR RMSE SAM SSIM SRTR 2.95 23.47 0.14 0.064 0.90 2.95 23.47 0.14 0.064 0.90 CSTF 4.64 20.64 0.19 0.074 0.75 4.32 20.57 0.20 0.073 0.79 BSR 4.15 20.05 0.21 0.074 0.87 4.15 20.05 0.21 0.074 0.87 CNMF 2.54 25.38 0.11 0.047 0.89 2.00 27.18 0.09 0.037 0.94 NLSTF 3.57 23.87 0.13 0.066 0.86 2.51 25.77 0.11 0.05 0.92 DHSR 2.43 27.62 0.08 0.039 0.92 1.76 29.35 0.06 0.031 0.95
下载: 导出CSV
表 3 在信噪比为35dB时算法在不同数据集上的运行时间
Table 3 Algorithm running time on different datasets at SNR of 35 dB
dataset time/s SRTR CSTF BSR CNMF NLSTF DHSR CAVE 206.48 144.39 621.44 307.86 585.94 98.63 Harvard 209.92 155.52 677.88 316.48 746.69 94.27
下载: 导出CSV
-
[1] 刘煊, 渠慎明. 低秩稀疏和改进SAM的高光谱图像误标签检测[J]. 激光技术, 2022, 46(6): 808-816. DOI: 10.7510/jgjs.issn.1001-3806.2022.06.016 LIU X, QU Sh M. False label detection in hyperspectral image based on low rank sparse and improved SAM[J]. Laser Technology, 2022, 46(6): 808-816(in Chinese). DOI: 10.7510/jgjs.issn.1001-3806.2022.06.016
[2] 臧传凯, 沈芳, 杨正东. 基于无人机高光谱遥感的河湖水环境探测[J]. 自然资源遥感, 2021, 33(3): 45-53. https://www.cnki.com.cn/Article/CJFDTOTAL-GTYG202103008.htm ZANG Ch K, SHEN F, YANG Zh D. Aquatic environmental monitoring of inland waters based on UAV hyperspectral remote sensing[J]. Remote Sensing for Natural Resources, 2021, 33(3): 45-53(in Chinese). https://www.cnki.com.cn/Article/CJFDTOTAL-GTYG202103008.htm
[3] 刘翠连, 陶于祥, 罗小波, 等. 混合卷积神经网络的高光谱图像分类方法[J]. 激光技术, 2022, 46(3): 355-361. DOI: 10.7510/jgjs.issn.1001-3806.2022.03.009 LIU C L, TAO Y X, LUO X B, et al. Hyperspectral image classification based on hybrid convolutional neural network[J]. Laser Techno-logy, 2022, 46(3): 355-361(in Chinese). DOI: 10.7510/jgjs.issn.1001-3806.2022.03.009
[4] YOKOYA N, YAIRI T, IWASAKI A. Coupled nonnegative matrix factorizationunmixing for hyperspectral and multispectral data fusion[J]. IEEE Transactions on Geoscience and Remote Sensing, 2011, 50(2): 528-537.
[5] YANG F X, PING Z L, MA F, et al. Fusion of hyperspectral and multispectral images with sparse and proximal regularization[J]. IEEE Access, 2019, 7: 186352-186363. DOI: 10.1109/ACCESS.2019.2961240
[6] AHMAD T, LYNGDOH R B, ANAND S S, et al. Robust coupled non-negative matrix factorization for hyperspectral and multispectral data fusion[C]//2021 IEEE International Geoscience and Remote Sensing Symposium IGARSS. New York, USA: IEEE Press, 2021: 2456-2459.
[7] KANATSOULIS C I, FU X, SIDIROPOULOS N D, et al. Hyperspectral super-resolution: A coupled tensor factorization approach[J]. IEEE Transactions on Signal Processing, 2018, 66(24): 6503-6517. DOI: 10.1109/TSP.2018.2876362
[8] MA F, HUO S, YANG F X. Graph-based logarithmic low-rank tensor decomposition for the fusion of remotely sensed images[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 11271-11286. DOI: 10.1109/JSTARS.2021.3123466
[9] HU J F, HUANG T Zh, DENG L J, et al. Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 33(12): 7251-7265.
[10] XIAO J J, LI J, YUAN Q Q, et al. Physics-based GAN with iterative refinement unit for hyperspectral and multispectral image fusion[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2021, 14: 6827-6841. DOI: 10.1109/JSTARS.2021.3075727
[11] HU J F, HUANG T Zh, DENG L J, et al. Fusformer: A transformer-based fusion network for hyperspectral image super-resolution[J]. IEEE Geoscience and Remote Sensing Letters, 2022, 19: 1-5.
[12] WU H P, XIAO B, CODELLA N, et al. Cvt: Introducing convolutions to vision transformers[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. New York, USA: IEEE Press, 2021: 22-31.
[13] HE A, LI T, LI N, et al. CABNet: Category attention block for imbalanced diabetic retinopathy grading[J]. IEEE Transactions on Medical Imaging, 2020, 40(1): 143-153.
[14] LI Zh W, SUI H, LUO C, et al. Morphological convolution and attention calibration network for hyperspectral and LiDAR data classification[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023, 16: 5728-5740. DOI: 10.1109/JSTARS.2023.3284655
[15] PAN H D, GAO F, DONG J Y, et al. Multiscale adaptive fusion network for hyperspectral image denoising[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2023, 16: 3045-3059. DOI: 10.1109/JSTARS.2023.3257051
[16] 王龙光, 郭裕兰, 林再平, 等. 基于Transformer的高光谱图像超分辨率重建[J]. 中国科学: 信息科学, 2023, 53(3): 500-516. https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202303006.htm WANG L G, GUO Y L, LIN Z P, et al. Deep hyperspectral image super-resolution with transformers[J]. Scientia Sinica: Informationis, 2023, 53(3): 500-516(in Chinese). https://www.cnki.com.cn/Article/CJFDTOTAL-PZKX202303006.htm
[17] SUI L Ch, LI L, LI J, et al. Fusion of hyperspectral and multispectral images based on a Bayesian nonparametric approach[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019, 12: 1205-1218. DOI: 10.1109/JSTARS.2019.2902847
[18] CHAI T, DRAXLER R R. Root mean square error (RMSE) or mean absolute error (MAE)-Arguments against avoiding RMSE in the literature[J]. Geoscientific Model Development, 2014, 7(3): 1247-1250. DOI: 10.5194/gmd-7-1247-2014
[19] JIANG J J, SUN H, LIU X M, et al. Learning spatial-spectral prior for super-resolution of hyperspectral imagery[J]. IEEE Transactions on Computational Imaging, 2020, 6: 1082-1096. DOI: 10.1109/TCI.2020.2996075
[20] LI Q, YUAN Y, JIA X P, et al. Dual-stage approach toward hyperspectral image super-resolution[J]. IEEE Transactions on Image Processing, 2022, 31: 7252-7263.
[21] WANG Zh, BOVIK A C, SHEIKH H R, et al. Image quality assessment: from error visibility to structural similarity[J]. IEEE Transactions on Image Processing, 2004, 13(4): 600-612.
[22] YASUMA F, MITSUNAGA T, ISO D, et al. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum[J]. IEEE Transactions on Image Processing, 2010, 19(9): 2241-2253.
[23] CHAKRABARTI A, ZICKLER T. Statistics of real-world hyperspectral images[C]//CVPR 2011. New York, USA: IEEE Press, 2011: 193-200.
[24] WEI Q, BIOUCAS-DIAS J, DOBIGEON N, et al. Hyperspectral and multispectral image fusion based on a sparse representation[J]. IEEE Transactions on Geoscience and Remote Sensing, 2015, 53(7): 3658-3668.
[25] LI Sh T, DIAN R W, FANG L Y, et al. Fusing hyperspectral and multispectral images via coupled sparse tensor factorization[J]. IEEE Transactions on Image Processing, 2018, 27(8): 4118-4130.
[26] DIAN R W, FANG L Y, LI Sh T. Hyperspectral image super-resolution via non-local sparse tensor factorization[C]//Proceedings of the IEEE Conference on Computer Vision and Pattem Recognition. New York, USA: IEEE Press, 2017: 5344-5353.
-
期刊类型引用(0)
其他类型引用(1)
计量
- 文章访问数: 18
- HTML全文浏览量: 7
- PDF下载量: 9
- 被引次数: 1