Malware detection

Business/Real-world Problem

What is Malware?

The term malware is a contraction of malicious software. Put simply, malware is any piece of software that was written with the intent of doing harm to data, devices or to people.
Source: https://www.avg.com/en/signal/what-is-malware

Problem Statement

In the past few years, the malware industry has grown very rapidly that, the syndicates invest heavily in technologies to evade traditional protection, forcing the anti-malware groups/communities to build more robust softwares to detect and terminate these attacks. The major part of protecting a computer system from a malware attack is to identify whether a given piece of file/software is a malware.

Source/Useful Links

Microsoft has been very active in building anti-malware products over the years and it runs it’s anti-malware utilities over 150 million computers around the world. This generates tens of millions of daily data points to be analyzed as potential malware. In order to be effective in analyzing and classifying such large amounts of data, we need to be able to group them into groups and identify their respective families.

This dataset provided by Microsoft contains about 9 classes of malware.

Source: https://www.kaggle.com/c/malware-classification

Real-world/Business objectives and constraints.

1. Minimize multi-class error.
2. Multi-class probability estimates.
3. Malware detection should not take hours and block the user's computer. It should fininsh in a few seconds or a minute.

Data Overview

Source : https://www.kaggle.com/c/malware-classification/data

For every malware, we have two files

1. .asm file (read more: https://www.reviversoft.com/file-extensions/asm)
2. .bytes file (the raw data contains the hexadecimal representation of the file's binary content, without the PE header)

Total train dataset consist of 200GB data out of which 50Gb of data is .bytes files and 150GB of data is .asm files:

Lots of Data for a single-box/computer.

There are total 10,868 .bytes files and 10,868 asm files total 21,736 files

There are 9 types of malwares (9 classes) in our give data

Types of Malware:

1. Ramnit
2. Lollipop
3. Kelihos_ver3
4. Vundo
5. Simda
6. Tracur
7. Kelihos_ver1
8. Obfuscator.ACY
9. Gatak

Mapping the real-world problem to an ML problem

There are nine different classes of malware that we need to classify a given a data point => Multi class classification problem

Performance Metric

Source: https://www.kaggle.com/c/malware-classification#evaluation Metric(s):

• Multi class log-loss
• Confusion matrix

Machine Learing Objectives and Constraints

Objective: Predict the probability of each data-point belonging to each of the nine classes.

Constraints:

• Class probabilities are needed
• Penalize the errors in class probabilites => Metric is Log-loss
• Some Latency constraints

Approcah

We first had to vectorize before feeding data to any model. For .bytes file, we created a vector starting from 0 to ff, basically a vocabulary of unique codes in all .bytes files to create a vector, a kind of bag of words

For .asm files, creaed a vocabulary like how we did in .bytes files(bag of words). Also used multiprogramming for faster computation

After getting the vectorized file, we experiemnted on both different vectors and also merged(.bytes and .asm) vectors

Results

Conclusion

This simple vectorization technique has really shown good results, We can try feature extraction and increase the data to make a good generalised model

Above results are using classical ML models, we can try with complex deep learning model in future