Analyzing Cross-domain Adversarial Vulnerabilities of Neural Networks for Image-based Malware Detection

Overview

This repository provides code and experiments for investigating adversarial attacks and generalization properties on deep learning models, specifically Convolutional Neural Networks (MobileNetV2), Vision Transformers (ViT-B/16), and architectures proposed in the literature, as Dual Convolutional Malware Network (DCMN) and GCSA-ResNet, applied to image-based malware detection and classification.

Adversarial examples pose serious security threats to machine learning models. Here, we develop various attack algorithms to evaluate the robustness of different architectures and study their generalization capabilities.

Citation

Important

To cite this work: coming soon

Repository Organization

MobileNetV2/

• Here you can find the Python code for the fine-tuning of the MobileNetV2 architecture for each dataset
• Under the subdirectory plots/ you can find all the plots associated with each architecture, under the name of the relative architecture (training and validation losses in .png files, performance metrics in .png files, confusion matrices in .png files, ROC curves in .png files, full metrics in .rtf files)

ViT-B:16/

• Here you can find the Python code for the fine-tuning of the ViT-B/16 architecture for each dataset
• Under the subdirectory plots/ you can find all the plots associated with each architecture, under the name of the relative architecture (training and validation losses in .png files, performance metrics in .png files, confusion matrices in .png files, ROC curves in .png files, full metrics in .rtf files)

DCMN/

• Here you can find the Python code for the training of the DCMN architecture for each dataset
• Under the subdirectory plots/ you can find all the plots associated with each architecture, under the name of the relative architecture (training and validation losses in .png files, performance metrics in .png files, confusion matrices in .png files, ROC curves in .png files, full metrics in .rtf files)

GCSA-ResNet/

• Here you can find the Python code for the training of the GCSA-ResNet architecture for each dataset
• Under the subdirectory plots/ you can find all the plots associated with each architecture, under the name of the relative architecture (training and validation losses in .png files, performance metrics in .png files, confusion matrices in .png files, ROC curves in .png files, full metrics in .rtf files)

Adversarial Attacks/

• Here you can find the Python code for each adversarial attack performed on each model
• Under the subdirectory pkl/ you can find the files collected for making the adversarial batch accuracy plots
• Under the subdirectory plots/ you can find all the plots associated with each attack (performance metrics in .png files, full metrics in .rtf files, example of the attack in .png files)

Generalization/

• Here you can find the Python code for the generalization study made on the detector models
• Under the subdirectory plots/ you can find all the plots associated with the generalization study (performance metrics in .png files, per-family metrics for BIG2015 dataset in .png files, confusion matrices in .png files, full metrics in .rtf files)

Acronyms of Adversarial Attacks

• FGSM = Fast Gradient Sign Method
• MI-FGSM = Magnitude Iterative FGSM
• PGD = Projected Gradient Descent
• APGD = Automatic PGD
• DF = DeepFool
• C&W = Carlini & Wagner
• SQ = Square
• XTI = eXecutable Transferring to Image

Overview of the Experiments

Architectures

Architecture	Type
MobileNetV2	CNN
ViT-B/16	Transformer
DCMN	ResNet-50 + CNN
GCSA-ResNet	GCSA + ResNet-50

Datasets

Dataset	Task	Samples	Training	Validation	Test	Distribution	Domain	Type	Classes
MalNet Tiny	Type Classification	87430	61201	8743	17486	70-10-20	Android	Byteplot	43
BIG2015	Family Classification	10868	6956	1736	2176	64-16-20	Windows	Byteplot	9
MaleX Balanced	Detection	359450	251645	35945	71890	70-10-20	Windows	Byteplot	2

Adversarial Attacks

Attack	Iterative	Technique	Type	Efficacy
FGSM	No	Gradient-based	White-box	Medium
MI-FGSM	Yes	Gradient-based	White-box	High
PGD	Yes	Gradient-based	White-box	Very High
APGD	Yes	Gradient-based	White-box	Very High
DeepFool	Yes	Boundary-based	White-box	High
Carlini&Wagner	Yes	Gradient-based	White-box	Very High
Square Attack	Yes	Random-based	Black-box	Medium

eXecutable Transferring to Image (XTI) Attack

We designed a novel adversarial attack to test the transferability of an adversarial attack performed in the binary domain and evaluated it on image domain models. The adversarial attack was inspired by the work of Bojan Kolosnjaji, Ambra Demontis, Battista Biggio, Davide Maiorca, Giorgio Giacinto, Claudia Eckert, and Fabio Roli. Adversarial malware binaries: Evading Deep Learning for Malware Detection in Executables, 2018.

Results of the Experiments

Summary of results

The results show that all the models are vulnerable to adversarial attacks, reaching to > 99% attack success rate. Moreover, we have demonstrated that some of the detection models also suffer from binary adversarial attacks. This study generally demonstrates the vulnerability of deep image-based malware classifiers and detectors to a broader spectrum of adversarial attacks.

Experiments Data

We report all the experimental results in the following attachment, providing a complete collection of data gathered from the training and fine-tuning process, the adversarial attacks results, and the generalization study: Experiments Data.xlsx.

Note

Some record comes from failed tests (e.g., for different parameters in the fine-tuning). The lines of failed experiments have been highlighted in gray.

The .xlsx file contains multiple sheets!