Heterogeneity-Aware Recurrent Neural Network for Hyperspectral and Multispectral Image Fusion

Due to the hardware limitations of remote imaging sensors, it is challenging to acquire images with high resolution in both the spatial and spectral domains. An effective and economical way to obtain high-resolution hyperspectral images (HR HSI) is to fuse low-resolution hyperspectral images (LR HSI) and high-resolution multispectral images (HR MSI). However, most existing deep learning based fusion methods employ the same network for all of the spectra without exploring their complex regional heterogeneity of hyperspectral characteristics. Taking various intrinsic spatial and spectral characteristics across different pixels into consideration, this paper proposes a mixture of recurrent neural networks (RNNs) under the variational probabilistic framework for spatial and spectral resolution enhancement. More specifically, we firstly cluster spectral characteristics into different groups, and employ different RNN experts for various spectra generation under the guidance of clustering. Moreover, a cluster-specific learnable Gaussian prior is proposed to provide a prior knowledge of heterogeneity. Further, an online variational inference scheme is derived for end-to-end optimization. Extensive experimental results demonstrate the effectiveness and efficiency of the proposed model on both synthetic and real datasets, compared with state-of-the-art unsupervised fusion methods.