Data Dimensional Reduction by Order Prediction in Heterogeneous Environment

P Suganya¹, Thirupurasundari D R²

^1,2Department of Computer Science and Engineering, Meenakshi College of Engineering, Tamil Nadu, Chennai.

Abstract

Equalizing the amount of processing time for each reducer instead of equalizing the amount of data each process in heterogeneous environment. A lightweight strategy to address the data skew problem among the reductions of MapReduce applications. MapReduce has been widely used in various applications, including web indexing, log analysis, data mining, scientific simulations and machine translations. The data skew refers to the imbalance in the amount of data assigned to each task.Using an innovative sampling method which can achieve a highly accurate approximation to the distribution of the intermediate data by sampling only a small fraction during the map processing and to reduce the data in reducer side. Prioritizing the sampling tasks for partitioning decision and splitting of large keys is supported when application semantics permit.Thus providing a reduced data of total ordered output as a result by range partitioner. In the proposed system, the data reduction is by predicting the reduction orders in parallel data processing using feature and instance selection. The accuracy of the data scale and data skew is effectively improved by CHI-ICF data reduction technique. In the existing system normal data distribution is calculated instead here still efficient distribution of data using the feature selection by ? 2 statistics (CHI) and instance selection by Iterative case filter (ICF) is processed.

The decision tree classifier is used to classify the data stream to produce an appropriate reduced data set.

Keywords

• MapReduce, data skew, sampling, partitioning, CHI-ICF, data reduction

Author's Profile

Author profile can be generated and linked through our partners World Book of Researchers. To include your profile online Click Here. After it is approved, please email to edlib @ asdf.res.in to create a link with all the papers.

• P Suganya : Profile
• Thirupurasundari D R : Profile

e-AID

Cite this Article as Follows

P Suganya, Thirupurasundari D R. "Data Dimensional Reduction by Order Prediction in Heterogeneous Environment" International Conference on Computer Applications (2016): 22-28. Print.