Design of Feature Selection Techniques and Classifiers for Big Data analysis

Abstract

The addition of knowledge and data has increased exponentially in the last decade or so. Previously only data entry operators were hired for uploading and entering useful information, but now with the advent of social networks and mobile internet, users are directly uploading semi structured and unstructured data on to the web. Each day some exabytes amount of data are being added to the world wide web. Traditional data mining techniques are facing huge computational and analytical challenges while handling these huge data. They are incompetent or inept in handling such huge volume of data, as they are not adapted to the new space and time requirements. Several cluster computing framework such as MapReduce, Spark and MPI are gradually gaining importance for handling and executing large scale data. Reducing the execution time has become the main focus of attention in this era of Big Data. Distributed computing is a method of processing large complex data using various parallel processing paradigm. MapReduce framework of Hadoop is one such cluster computing framework, which has gain fame for fast processing of Big Data, but it could not express same productivity for numerous iterative learning algorithms. A new cluster computing frame-work called Spark gives similar scalability and fault tolerance properties as MapReduce, and can run up to 100 times faster than Hadoop. We have proposed Spark based scalable statistical tests like ANOVA, Kruskal Wallis, Friedman, and information gain variants like Mutual Information and Mutual Information Feature Selection as well as Spark based scalable classifiers like SVM, Naive Bayes, Logistic Regression and Linear Regression for rapid processing and prediction of large datasets.