Big Data Analytics: Feature Selection and Machine Learning for Intrusion Detection on Microsoft Azure Platform

Abstract

In recent years, the overwhelming networking data has been growing at an exponential rate. Not only storage but also computing needs a system to process an intrusion detection system with a massive dataset. This research used cloud analytics to store big dataset, preprocess data, classify and evaluate results by using Microsoft azure, which can provide the appropriate environment. Because of the growth of data volume, intrusion detection model that adopts data mining technique has been used to detect intrusion pattern. Our research used mutual information and chi-square as a feature selection technique to reduce a feature set for computation time. Then, decision forest and neural network were used to classify the attack type of intrusion by 100% KDD CUP 1999 dataset. The performance of intrusion detection was measured by the accuracy of detection rate of attack type from the evaluation process in Microsoft azure.