Embedding compression with isotropic iterative quantization

Siyu Liao, Jie Chen, Yanzhi Wang, Qinru Qiu, Bo Yuan

Research output: Contribution to journalArticlepeer-review

Abstract

Continuous representation of words is a standard component in deep learning-based NLP models. However, representing a large vocabulary requires significant memory, which can cause problems, particularly on resource-constrained platforms. Therefore, in this paper we propose an isotropic iterative quantization (IIQ) approach for compressing embedding vectors into binary ones, leveraging the iterative quantization technique well established for image retrieval, while satisfying the desired isotropic property of PMI based models. Experiments with pre-trained embeddings (i.e., GloVe and HDC) demonstrate a more than thirty-fold compression ratio with comparable and sometimes even improved performance over the original real-valued embedding vectors.

Original languageEnglish (US)
JournalUnknown Journal
StatePublished - Jan 11 2020

ASJC Scopus subject areas

  • General

Fingerprint Dive into the research topics of 'Embedding compression with isotropic iterative quantization'. Together they form a unique fingerprint.

Cite this