TY - GEN
T1 - Theoretical properties for neural networks with weight matrices of low displacement rank
AU - Zhao, Liang
AU - Liao, Siyu
AU - Wang, Yanzhi
AU - Li, Zhe
AU - Tang, Jian
AU - Yuan, Bo
N1 - Publisher Copyright:
© Copyright 2017 by the authors(s).
PY - 2017
Y1 - 2017
N2 - Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.
AB - Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.
UR - http://www.scopus.com/inward/record.url?scp=85048550180&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85048550180&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85048550180
T3 - 34th International Conference on Machine Learning, ICML 2017
SP - 6186
EP - 6194
BT - 34th International Conference on Machine Learning, ICML 2017
PB - International Machine Learning Society (IMLS)
T2 - 34th International Conference on Machine Learning, ICML 2017
Y2 - 6 August 2017 through 11 August 2017
ER -