Heterogeneity-Aware Recurrent Neural Network for Hyperspectral and Multispectral Image Fusion

Ruiying Lu, Bo Chen, Jianqiao Sun, Wenchao Chen, Penghui Wang, Yuanwei Chen, Hongwei Liu, Pramod K. Varshney

Research output: Contribution to journalArticlepeer-review

2 Scopus citations


Due to the hardware limitations of remote imaging sensors, it is challenging to acquire images with high resolution in both the spatial and spectral domains. An effective and economical way to obtain high-resolution hyperspectral images (HR HSI) is to fuse low-resolution hyperspectral images (LR HSI) and high-resolution multispectral images (HR MSI). However, most existing deep learning based fusion methods employ the same network for all of the spectra without exploring their complex regional heterogeneity of hyperspectral characteristics. Taking various intrinsic spatial and spectral characteristics across different pixels into consideration, this paper proposes a mixture of recurrent neural networks (RNNs) under the variational probabilistic framework for spatial and spectral resolution enhancement. More specifically, we firstly cluster spectral characteristics into different groups, and employ different RNN experts for various spectra generation under the guidance of clustering. Moreover, a cluster-specific learnable Gaussian prior is proposed to provide a prior knowledge of heterogeneity. Further, an online variational inference scheme is derived for end-to-end optimization. Extensive experimental results demonstrate the effectiveness and efficiency of the proposed model on both synthetic and real datasets, compared with state-of-the-art unsupervised fusion methods.

Original languageEnglish (US)
Pages (from-to)649-665
Number of pages17
JournalIEEE Journal on Selected Topics in Signal Processing
Issue number4
StatePublished - Jun 1 2022


  • Dirichlet process mixture model
  • clustering
  • hyperspectral imaging
  • image fusion
  • probabilistic generative model
  • recurrent neural network

ASJC Scopus subject areas

  • Signal Processing
  • Electrical and Electronic Engineering


Dive into the research topics of 'Heterogeneity-Aware Recurrent Neural Network for Hyperspectral and Multispectral Image Fusion'. Together they form a unique fingerprint.

Cite this