A Design to Deal with Data Partition and Aggregation in Dynamic Manner

R Angayarkanni¹, U Karthick², V Hemalatha³, P Gomathi⁴

^1,2,3Assistant Professor/CSE, ⁴Dean, NSN College of Engineering and Technology, India

Abstract

Background: Map diminish job, we recollect combining knowledge with the same keys earlier than sending them too far off lessen tasks. Even though a similar function, called mix, has been already adopted via Hadoop, it operates abruptly after a map undertaking exclusively for its generated information, failing to exploit the information aggregation opportunities among multiple tasks on distinctive machines. Methods: We together consider data partition and aggregation for a Map curb job with a goal that is to diminish the complete network site visitors. In distinctive, we express an allotted algorithm for large knowledge applications by means of decomposing the long-established significant-scale trouble into a few sub issues that may be solved in parallel. Moreover, a web based algorithm is modeled to care for the data partition and aggregation in a dynamic manner. Findings: In the end, wide simulation outcome view our proposals can significantly scale back community traffic cost in each offline and on-line cases. Application: The MapReduce programming model simplifies large-scale data processing on commodity cluster by exploiting parallel map tasks and reduces tasks. Although many efforts have been made to improve the performance of MapReduce jobs, they ignore the network traffic generated in the shuffle phase, which plays a critical role in performance enhancement. Traditionally, a hash function is used to partition intermediate data among the Reduce tasks, which, however, is not traffic - efficient because network topology and data size associated with each key are not taken into consideration. In this paper, we study to reduce network traffic cost for a MapReduce job by designing a novel intermediate data partition scheme.

Keywords

Author's Profile

Author profile can be generated and linked through our partners World Book of Researchers. To include your profile online Click Here. After it is approved, please email to edlib @ asdf.res.in to create a link with all the papers.

• R Angayarkanni : Profile
• U Karthick : Profile
• V Hemalatha : Profile
• P Gomathi : Profile

e-AID

Cite this Article as Follows

R Angayarkanni, U Karthick, V Hemalatha, P Gomathi. "A Design to Deal with Data Partition and Aggregation in Dynamic Manner." International Conference on Intelligent Computing and Systems (2017): 21. Print.