A cross-job framework for MapReduce scheduling

Xuejie Xiao; Jian Tang; Zhenhua Chen; Jielong Xu; Chonggang Wang

doi:10.1109/BigData.2014.7004222

A cross-job framework for MapReduce scheduling

Xuejie Xiao, Jian Tang, Zhenhua Chen, Jielong Xu, Chonggang Wang

Department of Electrical Engineering & Computer Science

Research output: Chapter in Book/Entry/Poem › Conference contribution

2 Scopus citations

Abstract

In this paper, we present a novel cross-job framework for MapReduce scheduling, which aims to minimize the total processing time of a sequence of related jobs by combining reduce and map phases of two consecutive jobs and streaming data between them. The proposed framework has the following desirable properties: (1) It can accelerate the execution of a sequence of related MapReduce jobs by achieving a good tradeoff between data locality and parallelism. (2) It can support all the existing MapReduce applications with no changes to their source code. (3) It is a general framework, which can work with different scheduling algorithms. We built a new MapReduce runtime system called cross-job Hadoop by integrating the proposed cross-job framework into Hadoop. We conducted extensive experiments to evaluate its performance using PageRank and an Apache Pig application. Our experimental results show that the cross-job Hadoop can significantly reduce both the total processing time of a job sequence and the size of data transferred over the network.

Original language	English (US)
Title of host publication	Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014
Editors	Jimmy Lin, Jian Pei, Xiaohua Tony Hu, Wo Chang, Raghunath Nambiar, Charu Aggarwal, Nick Cercone, Vasant Honavar, Jun Huan, Bamshad Mobasher, Saumyadipta Pyne
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	135-140
Number of pages	6
ISBN (Electronic)	9781479956654
DOIs	https://doi.org/10.1109/BigData.2014.7004222
State	Published - 2014
Event	2nd IEEE International Conference on Big Data, IEEE Big Data 2014 - Washington, United States Duration: Oct 27 2014 → Oct 30 2014

Publication series

Name	Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014

Other

Other	2nd IEEE International Conference on Big Data, IEEE Big Data 2014
Country/Territory	United States
City	Washington
Period	10/27/14 → 10/30/14

Keywords

Big Data
MapReduce
Resource Management
Task Scheduling

ASJC Scopus subject areas

Artificial Intelligence
Information Systems

Access to Document

10.1109/BigData.2014.7004222

Cite this

Xiao, X., Tang, J., Chen, Z., Xu, J., & Wang, C. (2014). A cross-job framework for MapReduce scheduling. In J. Lin, J. Pei, X. T. Hu, W. Chang, R. Nambiar, C. Aggarwal, N. Cercone, V. Honavar, J. Huan, B. Mobasher, & S. Pyne (Eds.), Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014 (pp. 135-140). Article 7004222 (Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2014.7004222

A cross-job framework for MapReduce scheduling. / Xiao, Xuejie; Tang, Jian; Chen, Zhenhua et al.
Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014. ed. / Jimmy Lin; Jian Pei; Xiaohua Tony Hu; Wo Chang; Raghunath Nambiar; Charu Aggarwal; Nick Cercone; Vasant Honavar; Jun Huan; Bamshad Mobasher; Saumyadipta Pyne. Institute of Electrical and Electronics Engineers Inc., 2014. p. 135-140 7004222 (Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014).

Research output: Chapter in Book/Entry/Poem › Conference contribution

Xiao, X, Tang, J, Chen, Z, Xu, J & Wang, C 2014, A cross-job framework for MapReduce scheduling. in J Lin, J Pei, XT Hu, W Chang, R Nambiar, C Aggarwal, N Cercone, V Honavar, J Huan, B Mobasher & S Pyne (eds), Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014., 7004222, Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014, Institute of Electrical and Electronics Engineers Inc., pp. 135-140, 2nd IEEE International Conference on Big Data, IEEE Big Data 2014, Washington, United States, 10/27/14. https://doi.org/10.1109/BigData.2014.7004222

Xiao X, Tang J, Chen Z, Xu J, Wang C. A cross-job framework for MapReduce scheduling. In Lin J, Pei J, Hu XT, Chang W, Nambiar R, Aggarwal C, Cercone N, Honavar V, Huan J, Mobasher B, Pyne S, editors, Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014. Institute of Electrical and Electronics Engineers Inc. 2014. p. 135-140. 7004222. (Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014). doi: 10.1109/BigData.2014.7004222

Xiao, Xuejie ; Tang, Jian ; Chen, Zhenhua et al. / A cross-job framework for MapReduce scheduling. Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014. editor / Jimmy Lin ; Jian Pei ; Xiaohua Tony Hu ; Wo Chang ; Raghunath Nambiar ; Charu Aggarwal ; Nick Cercone ; Vasant Honavar ; Jun Huan ; Bamshad Mobasher ; Saumyadipta Pyne. Institute of Electrical and Electronics Engineers Inc., 2014. pp. 135-140 (Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014).

@inproceedings{a2580f470fcc4cce9182b43815fba74c,

title = "A cross-job framework for MapReduce scheduling",

abstract = "In this paper, we present a novel cross-job framework for MapReduce scheduling, which aims to minimize the total processing time of a sequence of related jobs by combining reduce and map phases of two consecutive jobs and streaming data between them. The proposed framework has the following desirable properties: (1) It can accelerate the execution of a sequence of related MapReduce jobs by achieving a good tradeoff between data locality and parallelism. (2) It can support all the existing MapReduce applications with no changes to their source code. (3) It is a general framework, which can work with different scheduling algorithms. We built a new MapReduce runtime system called cross-job Hadoop by integrating the proposed cross-job framework into Hadoop. We conducted extensive experiments to evaluate its performance using PageRank and an Apache Pig application. Our experimental results show that the cross-job Hadoop can significantly reduce both the total processing time of a job sequence and the size of data transferred over the network.",

keywords = "Big Data, MapReduce, Resource Management, Task Scheduling",

author = "Xuejie Xiao and Jian Tang and Zhenhua Chen and Jielong Xu and Chonggang Wang",

note = "Publisher Copyright: {\textcopyright} 2014 IEEE.; 2nd IEEE International Conference on Big Data, IEEE Big Data 2014 ; Conference date: 27-10-2014 Through 30-10-2014",

year = "2014",

doi = "10.1109/BigData.2014.7004222",

language = "English (US)",

series = "Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "135--140",

editor = "Jimmy Lin and Jian Pei and Hu, {Xiaohua Tony} and Wo Chang and Raghunath Nambiar and Charu Aggarwal and Nick Cercone and Vasant Honavar and Jun Huan and Bamshad Mobasher and Saumyadipta Pyne",

booktitle = "Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014",

}

TY - GEN

T1 - A cross-job framework for MapReduce scheduling

AU - Xiao, Xuejie

AU - Tang, Jian

AU - Chen, Zhenhua

AU - Xu, Jielong

AU - Wang, Chonggang

PY - 2014

Y1 - 2014

N2 - In this paper, we present a novel cross-job framework for MapReduce scheduling, which aims to minimize the total processing time of a sequence of related jobs by combining reduce and map phases of two consecutive jobs and streaming data between them. The proposed framework has the following desirable properties: (1) It can accelerate the execution of a sequence of related MapReduce jobs by achieving a good tradeoff between data locality and parallelism. (2) It can support all the existing MapReduce applications with no changes to their source code. (3) It is a general framework, which can work with different scheduling algorithms. We built a new MapReduce runtime system called cross-job Hadoop by integrating the proposed cross-job framework into Hadoop. We conducted extensive experiments to evaluate its performance using PageRank and an Apache Pig application. Our experimental results show that the cross-job Hadoop can significantly reduce both the total processing time of a job sequence and the size of data transferred over the network.

AB - In this paper, we present a novel cross-job framework for MapReduce scheduling, which aims to minimize the total processing time of a sequence of related jobs by combining reduce and map phases of two consecutive jobs and streaming data between them. The proposed framework has the following desirable properties: (1) It can accelerate the execution of a sequence of related MapReduce jobs by achieving a good tradeoff between data locality and parallelism. (2) It can support all the existing MapReduce applications with no changes to their source code. (3) It is a general framework, which can work with different scheduling algorithms. We built a new MapReduce runtime system called cross-job Hadoop by integrating the proposed cross-job framework into Hadoop. We conducted extensive experiments to evaluate its performance using PageRank and an Apache Pig application. Our experimental results show that the cross-job Hadoop can significantly reduce both the total processing time of a job sequence and the size of data transferred over the network.

KW - Big Data

KW - MapReduce

KW - Resource Management

KW - Task Scheduling

UR - http://www.scopus.com/inward/record.url?scp=84921721483&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84921721483&partnerID=8YFLogxK

U2 - 10.1109/BigData.2014.7004222

DO - 10.1109/BigData.2014.7004222

M3 - Conference contribution

AN - SCOPUS:84921721483

T3 - Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014

SP - 135

EP - 140

BT - Proceedings - 2014 IEEE International Conference on Big Data, IEEE Big Data 2014

A2 - Lin, Jimmy

A2 - Pei, Jian

A2 - Hu, Xiaohua Tony

A2 - Chang, Wo

A2 - Nambiar, Raghunath

A2 - Aggarwal, Charu

A2 - Cercone, Nick

A2 - Honavar, Vasant

A2 - Huan, Jun

A2 - Mobasher, Bamshad

A2 - Pyne, Saumyadipta

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2nd IEEE International Conference on Big Data, IEEE Big Data 2014

Y2 - 27 October 2014 through 30 October 2014

ER -

A cross-job framework for MapReduce scheduling

Abstract

Publication series

Other

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this