TY - GEN

T1 - On Distributed Stochastic Gradient Descent for Nonconvex Functions in the Presence of Byzantines

AU - Bulusu, Saikiran

AU - Khanduri, Prashant

AU - Sharma, Pranay

AU - Varshney, Pramod K.

N1 - Funding Information:
This work was supported in part by National Science Foundation under Grant ENG 60064237.

PY - 2020/5

Y1 - 2020/5

N2 - We consider the distributed stochastic optimization problem of minimizing a nonconvex function f in an adversarial setting. All the w worker nodes in the network are expected to send their stochastic gradient vectors to the fusion center (or server). However, some (at most α-fraction) of the nodes may be Byzantines, which may send arbitrary vectors instead. Vanilla implementation of distributed stochastic gradient descent (SGD) cannot handle such misbehavior from the nodes. We propose a robust variant of distributed SGD which is resilient to the presence of Byzantines. The fusion center employs a novel filtering rule that identifies and removes the Byzantine nodes. We show thatT = tilde Oleft( {frac{1}{{w{varepsilon 2}}} + frac{{{alpha 2}}}{{{varepsilon 2}}}} right) iterations are needed to achieve an-approximate stationary point (x such that{left| {nabla f(x)} right|2} leq varepsilon ) for the nonconvex learning problem. Unlike other existing approaches, the proposed algorithm is independent of the problem dimension.

AB - We consider the distributed stochastic optimization problem of minimizing a nonconvex function f in an adversarial setting. All the w worker nodes in the network are expected to send their stochastic gradient vectors to the fusion center (or server). However, some (at most α-fraction) of the nodes may be Byzantines, which may send arbitrary vectors instead. Vanilla implementation of distributed stochastic gradient descent (SGD) cannot handle such misbehavior from the nodes. We propose a robust variant of distributed SGD which is resilient to the presence of Byzantines. The fusion center employs a novel filtering rule that identifies and removes the Byzantine nodes. We show thatT = tilde Oleft( {frac{1}{{w{varepsilon 2}}} + frac{{{alpha 2}}}{{{varepsilon 2}}}} right) iterations are needed to achieve an-approximate stationary point (x such that{left| {nabla f(x)} right|2} leq varepsilon ) for the nonconvex learning problem. Unlike other existing approaches, the proposed algorithm is independent of the problem dimension.

KW - Adversarial machine learning

KW - Byzantines

KW - Distributed optimization

KW - Stochastic Gradient Descent

UR - http://www.scopus.com/inward/record.url?scp=85089235727&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85089235727&partnerID=8YFLogxK

U2 - 10.1109/ICASSP40776.2020.9052956

DO - 10.1109/ICASSP40776.2020.9052956

M3 - Conference contribution

AN - SCOPUS:85089235727

T3 - ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings

SP - 3137

EP - 3141

BT - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2020 IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2020

Y2 - 4 May 2020 through 8 May 2020

ER -