Novel Approach for Data Encryption in Hadoop

Abstract

The data used by various organizations and institutions is increasing at a rapid rate. In todays world, organizations have to process petabytes of data. The traditional database management system fails to process such large amount of data. So, we need to find out an effective way of approach for handling and processing such large amount of data which gives rise to big data problem. Size or volume of the data is not only the single criteria to classify big data, we have to keep in mind the type of the data that is whether data is structured or semistructured or unstructured. Semistructured or unstructured data is difficult to manage through traditional database management system. To overcome this big data problem Apache produced a software tool named Hadoop which uses MapReduce architecture to process large amount of data. MapReduce architecture processes the data in parallel manner. Hence, we have analyzed the performance of Hadoop with increase in size of the dataset. We have analyzed how Hadoop’s performance varies with increase in size of the dataset. Though it gives solution to overcome the big data problem but it does not ensure the security and privacy of the stored files in Hadoop. In this dissertation, an asymmetric key cryptosystem is proposed for the encryption of files stored in HDFS. We used our proposed cryptosystem to encrypt the content of data before storing it in HDFS. The results obtained from various experiments indicate favorable results of above approach to address security problem associated with big data.