StructADMM: Achieving Ultrahigh Efficiency in Structured Pruning for DNNs

Tianyun Zhang, Shaokai Ye, Xiaoyu Feng, Xiaolong Ma, Kaiqi Zhang, Zhengang Li, Jian Tang, Sijia Liu, Xue Lin, Yongpan Liu, Makan Fardad, Yanzhi Wang

Research output: Contribution to journalArticlepeer-review

Abstract

Weight pruning methods of deep neural networks (DNNs) have been demonstrated to achieve a good model pruning rate without loss of accuracy, thereby alleviating the significant computation/storage requirements of large-scale DNNs. Structured weight pruning methods have been proposed to overcome the limitation of irregular network structure and demonstrated actual GPU acceleration. However, in prior work, the pruning rate (degree of sparsity) and GPU acceleration are limited (to less than 50%) when accuracy needs to be maintained. In this work, we overcome these limitations by proposing a unified, systematic framework of structured weight pruning for DNNs. It is a framework that can be used to induce different types of structured sparsity, such as filterwise, channelwise, and shapewise sparsity, as well as nonstructured sparsity. The proposed framework incorporates stochastic gradient descent (SGD; or ADAM) with alternating direction method of multipliers (ADMM) and can be understood as a dynamic regularization method in which the regularization target is analytically updated in each iteration. Leveraging special characteristics of ADMM, we further propose a progressive, multistep weight pruning framework and a network purification and unused path removal procedure, in order to achieve higher pruning rate without accuracy loss. Without loss of accuracy on the AlexNet model, we achieve 2.58x and 3.65x average measured speedup on two GPUs, clearly outperforming the prior work. The average speedups reach 3.15x and 8.52x when allowing a moderate accuracy loss of 2%. In this case, the model compression for convolutional layers is 15.0x, corresponding to 11.93x measured CPU speedup. As another example, for the ResNet-18 model on the CIFAR-10 data set, we achieve an unprecedented 54.2x structured pruning rate on CONV layers. This is 32x higher pruning rate compared with recent work and can further translate into 7.6x inference time speedup on the Adreno 640 mobile GPU compared with the original, unpruned DNN model. We share our codes and models at the link http://bit.ly/2M0V7DO.

Original languageEnglish (US)
JournalIEEE Transactions on Neural Networks and Learning Systems
DOIs
StateAccepted/In press - 2021

Keywords

  • Acceleration
  • Alternating direction method of multipliers (ADMM)
  • Convex functions
  • Degradation
  • Graphics processing units
  • Optimization
  • Periodic structures
  • Quantization (signal)
  • deep neural networks (DNNs)
  • hardware acceleration
  • weight pruning.

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'StructADMM: Achieving Ultrahigh Efficiency in Structured Pruning for DNNs'. Together they form a unique fingerprint.

Cite this