SlimML: Removing Non-Critical Input Data in Large-Scale Iterative Machine Learning

Rui Han, Chi Harold Liu, Shilin Li, Lydia Y. Chen, Guoren Wang, Jian Tang, Jieping Ye

Research output: Contribution to journalArticlepeer-review

2 Scopus citations

Abstract

The core of many large-scale machine learning (ML) applications, such as neural networks (NN), support vector machine (SVM), and convolutional neural network (CNN), is the training algorithm that iteratively updates model parameters by processing massive datasets. From a plethora of studies aiming at accelerating ML, being data parallelization and parameter server, the prevalent assumption is that all data points are equivalently relevant to model parameter updating. In this article, we challenge this assumption by proposing a criterion to measure a data point's effect on model parameter updating, and experimentally demonstrate that the majority of data points are non-critical in the training process. We develop a slim learning framework, termed SlimML, which trains the ML models only on the critical data and thus significantly improves training performance. To such an end, SlimML efficiently leverages a small number of aggregated data points per iteration to approximate the criticalness of original input data instances. The proposed approach can be used by changing a few lines of code in a standard stochastic gradient descent (SGD) procedure, and we demonstrate experimentally, on NN regression, SVM classification, and CNN training, that for large datasets, it accelerates model training process by an average of 3.61 times while only incurring accuracy losses of 0.37 percent.

Original languageEnglish (US)
Article number8890886
Pages (from-to)2223-2236
Number of pages14
JournalIEEE Transactions on Knowledge and Data Engineering
Volume33
Issue number5
DOIs
StatePublished - May 1 2021

Keywords

  • Iterative machine learning
  • MapReduce
  • large input datasets
  • model parameter updating

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics

Fingerprint

Dive into the research topics of 'SlimML: Removing Non-Critical Input Data in Large-Scale Iterative Machine Learning'. Together they form a unique fingerprint.

Cite this