added HyDiff experimental subjects, result summaries, scripts, tool, …

…Dockerfile, and License

.dockerignore

@@ -0,0 +1 @@
+build

Dockerfile

@@ -0,0 +1,18 @@
+FROM ubuntu:18.04
+
+MAINTAINER Yannic Noller <yannic.noller@acm.org>
+
+# Dependencies
+RUN apt-get -y update
+RUN apt-get -y install git build-essential openjdk-8-jdk wget unzip ant python3 python3-numpy vim nano
+RUN update-java-alternatives --set /usr/lib/jvm/java-1.8.0-openjdk-amd64
+RUN wget https://services.gradle.org/distributions/gradle-4.4.1-bin.zip -P /tmp
+RUN unzip -d /opt/gradle /tmp/gradle-*.zip
+ENV GRADLE_HOME=/opt/gradle/gradle-4.4.1
+ENV PATH=${GRADLE_HOME}/bin:${PATH}
+
+# Installing HyDiff
+WORKDIR /root
+RUN git clone https://github.com/yannicnoller/hydiff.git --branch v1.0.0
+WORKDIR /root/hydiff
+RUN ./setup.sh

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2020 Yannic Noller, Corina S. Pasareanu, Marcel Böhme, Youcheng Sun, Hoang Lam Nguyen, Lars Grunske
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,4 +1,141 @@
-
+<!-- [![DOI](https://zenodo.org/badge/157902250.svg)](https://zenodo.org/badge/latestdoi/157902250) -->
 # HyDiff: Hybrid Differential Software Analysis
+This repository provides the tool and the evaluation subjects for the paper *HyDiff: Hybrid Differential Software Analysis* accepted for the technical track at ICSE'2020. A pre-print of the paper is available [here](https://yannicnoller.github.io/publications/icse2020_noller_hydiff.pdf).
+
+Authors:
+[Yannic Noller](https://yannicnoller.github.io),
+[Corina S. Pasareanu](https://www.cylab.cmu.edu/directory/bios/pasareanu-corina.html),
+[Marcel Böhme](https://mboehme.github.io),
+[Youcheng Sun](https://sites.google.com/site/theyoucheng/),
+[Hoang Lam Nguyen](https://github.com/hoanglam-nguyen),
+and [Lars Grunske](https://www.informatik.hu-berlin.de/de/Members/lars-grunske).
+
+The repository includes:
+* a [setup](setup.sh) script,
+* the experiment subjects: [experiments/subjects](./experiments/subjects),
+* the summarized experiment results: [experiments/results](./experiments/results),
+* the scripts to rerun all experiments: [experiments/scripts](./experiments/scripts),
+* and the source code for both components of Hydiff: [tool/fuzzing](./tool/fuzzing), and [tool/symbolicexecution](./tool/symbolicexecution).
+
+A pre-built version of HyDiff is also available as [Docker image](https://hub.docker.com/r/yannicnoller/hydiff):
+```
+docker pull yannicnoller/hydiff
+docker run -it --rm yannicnoller/hydiff
+```
+
+## Tool
+HyDiff's technical framework is built on top of [Badger](https://github.com/isstac/badger), [DifFuzz](https://github.com/isstac/diffuzz), and the [Symbolic PathFinder](https://github.com/SymbolicPathFinder).
+We provide a complete snapshot of all tools and our extensions.
+
+### Requirements
+* Git, Ant, Build-Essentials, Gradle
+* Java JDK = 1.8
+* Python3, Numpy Package
+* recommended: Ubuntu 18.04.1 LTS
+
+### Folder Structure
+The folder *tool* contains 2 subfolders: *fuzzing* and *symbolicexecution*, representing the both components of HyDiff.
+
+#### fuzzing
+
+* *afl-differential*:
+The fuzzing component is built on top of DifFuzz and KelinciWCA (the fuzzing part of Badger).
+Both use [AFL](http://lcamtuf.coredump.cx/afl/) as the underlying fuzzing engine.
+In order to make it easy for the users, we provide our complete modified AFL variant in this folder.
+Our modifications are based on [afl-2.52b](http://lcamtuf.coredump.cx/afl/releases/?O=D).
+
+* *kelinci-differential*: 
+Kelinci leverages a server-client architecture to make AFL applicable to Java applications, please refer to the Kelinci [poster-paper](https://dl.acm.org/citation.cfm?id=3138820) for more details.
+We modified it to make usable in a general differential analysis.
+It includes an *interface* program to connect the *Kelinci server* to the AFL fuzzer and the *instrumentor* project, which is used to instrument the Java bytecode.
+The instrumentation handles the coverage reporting and the collection of our differential metrics.
+The Kelinci server handles requests from AFL to execute a mutated input on the application.
+
+#### symbolicexecution
+
+* *jpf-core*:
+Our symbolic execution is built on top of Symbolic PathFinder (SPF), which is an extension of [Java PathFinder](https://github.com/javapathfinder) (JPF), which makes it necessary to include the core implementation of JPF.
+
+* *jpf-symbc-differential*:
+In order to make SPF applicable to a differential analysis, we modified in several locations and added the ability to perform some sort of *shadow symbolic execution* (cf. [Complete Shadow Symbolic Execution with Java PathFinder](https://github.com/hub-se/jpf-shadow-plus)).
+This folder includes the modified SPF project.
+
+* *badger-differential*:
+HyDiff performs a hybrid analysis by running fuzzing and symbolic execution in parallel.
+This concept is based on Badger, which provides the technical basis for our implementation.
+This folder includes the modified Badger project, which enables the differential hybrid analysis, incl. the differential dynamic symbolic execution.
+
+### How to install the tool and run our evaluation
+Be aware that the instructions have been tested for Unix systems only.
+
+1. First you need to build the tool and the subjects.
+We provide a script *setup.sh* to simply build everything.
+Note: the script may override an existing site.properties file, which is required for JPF/SPF.
+
+2. Test the installation: the best way to test the installation is to execute the evaluation of our example program (cf. Listing 1 in our paper).
+You can execute the script [run_example.sh](./experiments/scripts/run_example.sh).
+As it is, it will run each analysis (just differential fuzzing, just differential symbolic execution, and the hybrid analysis) once.
+The values presented in our paper in Section 2.2 are averaged over 30 runs.
+In order to perform 30 runs each, you can easily adapt the script, but for some first test runs you can leave it as it is.
+The script should produce three folders:
+    * experiments/subjects/example/fuzzer-out-1: results for differential fuzzing
+    * experiments/subjects/example/symexe-out-1: results for differential symbolic execution
+    * experiments/subjects/example/hydiff-out-1: results for HyDiff (hybrid combination)
+It will also produce three csv files with the summarized statistics for each experiment:
+    * experiments/subjects/example/fuzzer-out-results-n=1-t=600-s=30.csv
+    * experiments/subjects/example/symexe-out-results-n=1-t=600-s=30.csv
+    * experiments/subjects/example/hydiff-out-results-n=1-t=600-s=30-d=0.csv
+
+3. After finishing the building process and testing the installation, you can use the provided *run* scripts ([experiments/scripts](./experiments/scripts)) to replay HyDiff's evaluation or to perform your own differential analysis.
+HyDiff's evaluation contains three types of differential analysis.
+For each of them you will find a separate run script:
+    * [run_regression_evaluation.sh](./experiments/scripts/run_regression_evaluation.sh)
+    * [run_sidechannel_evaluation.sh](./experiments/scripts/run_sidechannel_evaluation.sh)
+    * [run_dnn_evaluation.sh](./experiments/scripts/run_dnn_evaluation.sh)
+
+In the beginning of each run script you can define the experiment parameters:
+* `number_of_runs`: `N`, the number of evaluation runs for each subject (30 for all experiments)
+* `time_bound`: `T`, the time bound for the analysis (regression: 600sec, side-channel: 1800sec, and dnn: 3600sec)
+* `step_size_eval`: `S`, the step size for the evaluation (30sec for all experiments)
+* [`time_symexe_first`: `D`, the delay with which fuzzing gets started after symexe for the DNN subjects] (only DNN)
+
+Each run script first executes differential fuzzing, then differential symbolic execution and then the hybrid analysis.
+Please adapt our scripts to perform your own analysis.
+
+For each *subject*, *analysis_type*, and experiment repetition *i* the scripts will produce folders like:
+`experiments/subjects/<subject>/<analysis_type>-out-<i>`,
+and will summarize the experiments in csv files like:
+`experiments/subjects/<subject>/<analysis_type>-out-results-n=<N>-t=<T>-s=<S>-d=<D>.csv`.
+
+### Complete Evaluation Reproduction
+In order to reproduce our evaluation completely, you need to run the three mentioned run scripts.
+They include the generation of all statistics.
+Be aware that the mere runtime of all analysis parts is more than **53 days** because of the high runtimes and number of repetitions.
+So it might be worthwhile to run it only for some specific subjects or to run the analysis on different machines in parallel or to modify the runtime or to reduce the number of repetitions.
+Feel free to adjust the script or reuse it for your own purpose.
+
+### Statistics
+As mentioned earlier, the statistics will be automatically generated by our run script, which execute the python scripts from the *scripts* folder to aggregate the several experiment runs.
+They will generate csv files with the information about the average result values.
+
+For the regression analysis and the DNN analysis we use the scripts:
+* [experiments/scripts/evaluate_regression_fuzz.py](./experiments/scripts/evaluate_regression_fuzz.py)
+* [experiments/scripts/evaluate_regression_symexe.py](./experiments/scripts/evaluate_regression_symexe.py)
+* [experiments/scripts/evaluate_regression_hydiff.py](./experiments/scripts/evaluate_regression_hydiff.py)
+
+For the side-channel analysis we use the scripts:
+* [experiments/scripts/evaluate_cost_fuzz.py](./experiments/scripts/evaluate_cost_fuzz.py)
+* [experiments/scripts/evaluate_cost_symexe.py](./experiments/scripts/evaluate_cost_symexe.py)
+* [experiments/scripts/evaluate_cost_hydiff.py](./experiments/scripts/evaluate_cost_hydiff.py)
+
+All csv files for our experiments are included in [experiments/results](./experiments/results).
+
+Feel free to adapt these evaluation scripts for your own purpose.
+
+## Maintainers
+
+* **Yannic Noller** (yannic.noller at acm.org)
+
 
-(in progress)
+## License
+This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details

experiments/results/dnn/mnist2_1/fuzzer-out-results-n=30-t=3600-s=30.csv

@@ -0,0 +1,127 @@
+seconds,avg_odiff,ci_odiff,avg_ddiff,ci_ddiff
+30,0.00,0.00,0.00,0.00
+60,0.00,0.00,0.00,0.00
+90,0.00,0.00,0.00,0.00
+120,0.00,0.00,0.00,0.00
+150,0.00,0.00,0.00,0.00
+180,0.00,0.00,0.00,0.00
+210,0.00,0.00,0.00,0.00
+240,0.00,0.00,0.00,0.00
+270,0.00,0.00,0.00,0.00
+300,0.00,0.00,0.00,0.00
+330,0.00,0.00,0.00,0.00
+360,0.00,0.00,0.00,0.00
+390,0.00,0.00,0.00,0.00
+420,0.00,0.00,0.00,0.00
+450,0.00,0.00,0.00,0.00
+480,0.00,0.00,0.00,0.00
+510,0.00,0.00,0.00,0.00
+540,0.00,0.00,0.00,0.00
+570,0.00,0.00,0.00,0.00
+600,0.00,0.00,0.00,0.00
+630,0.00,0.00,0.00,0.00
+660,0.00,0.00,0.00,0.00
+690,0.00,0.00,0.00,0.00
+720,0.00,0.00,0.03,0.06
+750,0.00,0.00,0.03,0.06
+780,0.00,0.00,0.03,0.06
+810,0.00,0.00,0.03,0.06
+840,0.00,0.00,0.03,0.06
+870,0.00,0.00,0.03,0.06
+900,0.00,0.00,0.03,0.06
+930,0.00,0.00,0.03,0.06
+960,0.00,0.00,0.03,0.06
+990,0.00,0.00,0.03,0.06
+1020,0.00,0.00,0.03,0.06
+1050,0.00,0.00,0.43,0.18
+1080,0.03,0.06,0.73,0.16
+1110,0.03,0.06,0.90,0.14
+1140,0.03,0.06,0.97,0.11
+1170,0.03,0.06,1.00,0.09
+1200,0.07,0.09,1.03,0.06
+1230,0.07,0.09,1.03,0.06
+1260,0.07,0.09,1.03,0.06
+1290,0.07,0.09,1.03,0.06
+1320,0.07,0.09,1.03,0.06
+1350,0.07,0.09,1.03,0.06
+1380,0.13,0.12,1.20,0.14
+1410,0.17,0.13,1.50,0.18
+1440,0.17,0.13,1.60,0.18
+1470,0.17,0.13,1.83,0.16
+1500,0.20,0.14,1.97,0.11
+1530,0.20,0.14,2.03,0.06
+1560,0.20,0.14,2.03,0.06
+1590,0.20,0.14,2.03,0.06
+1620,0.20,0.14,2.03,0.06
+1650,0.20,0.14,2.03,0.06
+1680,0.20,0.14,2.03,0.06
+1710,0.20,0.14,2.07,0.09
+1740,0.20,0.14,2.27,0.16
+1770,0.23,0.15,2.50,0.18
+1800,0.27,0.16,2.77,0.18
+1830,0.30,0.16,2.83,0.16
+1860,0.30,0.16,2.93,0.13
+1890,0.30,0.16,2.93,0.13
+1920,0.30,0.16,3.03,0.06
+1950,0.30,0.16,3.03,0.06
+1980,0.30,0.16,3.03,0.06
+2010,0.30,0.16,3.03,0.06
+2040,0.30,0.16,3.03,0.06
+2070,0.30,0.16,3.13,0.12
+2100,0.33,0.17,3.40,0.18
+2130,0.40,0.18,3.60,0.20
+2160,0.40,0.18,3.63,0.20
+2190,0.40,0.18,3.83,0.16
+2220,0.40,0.18,3.90,0.14
+2250,0.40,0.18,3.97,0.11
+2280,0.40,0.18,4.03,0.06
+2310,0.40,0.18,4.03,0.06
+2340,0.40,0.18,4.03,0.06
+2370,0.40,0.18,4.03,0.06
+2400,0.40,0.18,4.13,0.12
+2430,0.43,0.18,4.27,0.16
+2460,0.43,0.18,4.43,0.20
+2490,0.43,0.18,4.53,0.20
+2520,0.43,0.18,4.67,0.19
+2550,0.43,0.18,4.77,0.18
+2580,0.43,0.18,4.87,0.15
+2610,0.43,0.18,5.03,0.06
+2640,0.43,0.18,5.03,0.06
+2670,0.43,0.18,5.03,0.06
+2700,0.43,0.18,5.03,0.06
+2730,0.43,0.18,5.10,0.11
+2760,0.43,0.18,5.13,0.12
+2790,0.43,0.18,5.20,0.14
+2820,0.43,0.18,5.47,0.20
+2850,0.43,0.18,5.53,0.20
+2880,0.43,0.18,5.63,0.20
+2910,0.43,0.18,5.73,0.18
+2940,0.43,0.18,5.90,0.14
+2970,0.47,0.18,6.00,0.09
+3000,0.47,0.18,6.03,0.06
+3030,0.47,0.18,6.03,0.06
+3060,0.47,0.18,6.10,0.11
+3090,0.47,0.18,6.10,0.11
+3120,0.47,0.18,6.13,0.12
+3150,0.47,0.18,6.27,0.18
+3180,0.47,0.18,6.40,0.20
+3210,0.50,0.18,6.53,0.20
+3240,0.53,0.18,6.67,0.19
+3270,0.53,0.18,6.83,0.16
+3300,0.53,0.18,6.90,0.14
+3330,0.57,0.20,7.00,0.09
+3360,0.57,0.20,7.03,0.06
+3390,0.57,0.20,7.03,0.06
+3420,0.57,0.20,7.07,0.09
+3450,0.57,0.20,7.10,0.11
+3480,0.57,0.20,7.20,0.14
+3510,0.57,0.20,7.27,0.16
+3540,0.57,0.20,7.33,0.17
+3570,0.57,0.20,7.50,0.20
+3600,0.57,0.20,7.73,0.18
+
+time +odiff>0:
+2725.40 (+/- 341.09) min=1074
++odiff_times=[1375, 1373, 3600, 3600, 3210, 3600, 1396, 3600, 2430, 1776, 3600, 3600, 3600, 3600, 1748, 1485, 3600, 2073, 1185, 2120, 3600, 3600, 3600, 2959, 3600, 3600, 1074, 3230, 1808, 2120]
+#odiffs=[1, 1, 0, 0, 1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 2, 1]
+#ddiffs=[8, 8, 8, 8, 8, 8, 8, 7, 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 7, 8, 8, 8, 8, 7, 8, 8, 7, 8, 7, 9]