Skip to content

Commit 7027ec3

Browse files
fjwillemsenfjwillemsen
committed
Updated schema and README
1 parent 8ca3a4b commit 7027ec3

File tree

11 files changed

+165
-129
lines changed

11 files changed

+165
-129
lines changed

‎README.md‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,10 @@
1212

1313

1414
This repository contains the software package accompanying the paper "A Methodology for Comparing Auto-Tuning Optimization Algorithms".
15-
It makes the guidelines in the methodology easy to apply: simply specify the `.json` file, run it with `python visualize_experiments.py` and wait for the results!
15+
It makes the guidelines in the methodology easy to apply: simply specify the `.json` file, run `autotuning_visualize [path_to_json]` and wait for the results!
1616

1717
## Installation
18-
The package can be installed by cloning this repository and running `pip install .`. Python >= 3.9 is supported.
18+
The package can be installed by cloning this repository and running `pip install .`. Python >= 3.9 is supported.
1919

2020
## Notable features
2121
- Official software by the authors of the methodology-defining paper.
@@ -28,6 +28,23 @@ The package can be installed by cloning this repository and running `pip install
2828
<img width="674" alt="example run in interactive window" src="https://user-images.githubusercontent.com/6725103/232880006-70a05b0e-a4e4-4cc7-bea9-473959c474c2.png">
2929
<img width="483" alt="example run in interactive window 2" src="https://user-images.githubusercontent.com/6725103/232881244-d432ea8e-801a-44b1-9acb-b98cc1b740ac.png">
3030

31+
## Usage
32+
33+
### Entry points
34+
There are two entry points defined: `autotuning_experiment` and `autotuning_visualize`. Both take one argument: the path to an experiment file (see below).
35+
36+
### Input files
37+
To get started, all you need is an experiments file. This is a `json` file that describes the details of your comparison: which algorithms to use, which programs to tune on which devices, the graphs to output and so on.
38+
You can find the API and an example `experiments.json` in the [documentation]().
39+
40+
### File references
41+
As we are dealing with input and output files, file references matter.
42+
When calling the entrypoints, we are already providing the path to an experiments file.
43+
File references in experiments files are relative to the location of the experiment file itself.
44+
File references in tuning scripts are relative to the location of the tuning script itself. Tuning scripts need to have the global literals `file_path_results` and `file_path_metadata` for this package to know where to get the results.
45+
Plots outputted by this package are placed in a folder called `generated_plots` relative to the current working directory.
46+
47+
3148
## Contributing
3249

3350
### Setup

‎experiment_files/methodology_paper_example.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"version": "0.1.2",
33
"name": "Methodology paper evaluation",
44
"folder_id": "methodology_paper_evaluation",
5-
"kernel_path": "../cached_data_used/kernels",
5+
"kernels_path": "../cached_data_used/kernels",
66
"bruteforced_caches_path": "../cached_data_used/cachefiles",
77
"visualization_caches_path": "../cached_data_used/visualizations",
88
"kernels": [

‎experiment_files/presentation_comparison.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"version": "0.1.2",
33
"name": "Example comparison for presentation",
44
"folder_id": "presentation_comparison",
5-
"kernel_path": "cached_data_used/kernels",
5+
"kernels_path": "cached_data_used/kernels",
66
"kernels": [
77
"pnpoly",
88
"convolution"
@@ -68,4 +68,4 @@
6868
}
6969
}
7070
]
71-
}
71+
}

‎experiment_files/test_output_file_writer.json‎

Lines changed: 58 additions & 58 deletions
Original file line numberDiff line numberDiff line change
@@ -1,63 +1,63 @@
11
{
2-
"version": "0.1.2",
3-
"name": "Test output file writer",
4-
"folder_id": "test_output_file_writer",
5-
"kernels": [
6-
"convolution"
7-
],
8-
"kernel_path": "../cached_data_used/kernels",
9-
"bruteforced_caches_path": "../cached_data_used/cachefiles",
10-
"visualization_caches_path": "../cached_data_used/visualizations",
11-
"GPUs": [
12-
"RTX_2080_Ti"
13-
],
14-
"minimization": true,
15-
"resolution": 1e3,
16-
"cutoff_percentile": 0.99,
17-
"cutoff_percentile_start": 0.7,
18-
"objective_time_keys": [
19-
"compilation",
20-
"benchmark",
21-
"framework",
22-
"search_algorithm",
23-
"validation"
24-
],
25-
"objective_performance_keys": [
26-
"time"
27-
],
28-
"cutoff_type": "fevals",
29-
"plot": {
30-
"plot_x_value_types": [
31-
"aggregated"
2+
"version": "0.1.2",
3+
"name": "Test output file writer",
4+
"folder_id": "test_output_file_writer",
5+
"kernels": [
6+
"convolution"
327
],
33-
"plot_y_value_types": [
34-
"normalized",
35-
"baseline"
8+
"kernels_path": "../cached_data_used/kernels",
9+
"bruteforced_caches_path": "../cached_data_used/cachefiles",
10+
"visualization_caches_path": "../cached_data_used/visualizations",
11+
"GPUs": [
12+
"RTX_2080_Ti"
3613
],
37-
"confidence_level": 0.95,
38-
"compare_baselines": true,
39-
"compare_split_times": true
40-
},
41-
"strategy_defaults": {
42-
"repeats": 100,
43-
"minimum_number_of_evaluations": 20,
44-
"stochastic": true,
45-
"record_data": [
46-
"time",
47-
"GFLOP/s"
48-
]
49-
},
50-
"strategies": [
51-
{
52-
"name": "random_sample_100_iter",
53-
"strategy": "random_sample",
54-
"display_name": "Random sampling 100 iters",
55-
"repeats": 5
14+
"minimization": true,
15+
"resolution": 1e3,
16+
"cutoff_percentile": 0.99,
17+
"cutoff_percentile_start": 0.7,
18+
"objective_time_keys": [
19+
"compilation",
20+
"benchmark",
21+
"framework",
22+
"search_algorithm",
23+
"validation"
24+
],
25+
"objective_performance_keys": [
26+
"time"
27+
],
28+
"cutoff_type": "fevals",
29+
"plot": {
30+
"plot_x_value_types": [
31+
"aggregated"
32+
],
33+
"plot_y_value_types": [
34+
"normalized",
35+
"baseline"
36+
],
37+
"confidence_level": 0.95,
38+
"compare_baselines": true,
39+
"compare_split_times": true
5640
},
57-
{
58-
"name": "genetic_algorithm",
59-
"strategy": "genetic_algorithm",
60-
"display_name": "Genetic Algorithm"
61-
}
62-
]
41+
"strategy_defaults": {
42+
"repeats": 100,
43+
"minimum_number_of_evaluations": 20,
44+
"stochastic": true,
45+
"record_data": [
46+
"time",
47+
"GFLOP/s"
48+
]
49+
},
50+
"strategies": [
51+
{
52+
"name": "random_sample_100_iter",
53+
"strategy": "random_sample",
54+
"display_name": "Random sampling 100 iters",
55+
"repeats": 5
56+
},
57+
{
58+
"name": "genetic_algorithm",
59+
"strategy": "genetic_algorithm",
60+
"display_name": "Genetic Algorithm"
61+
}
62+
]
6363
}

‎experiment_files/test_random_calculated.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
"version": "0.1.2",
33
"name": "Test random search (calculated vs executed)",
44
"folder_id": "test_random_calculated",
5-
"kernel_path": "../cached_data_used/kernels",
5+
"kernels_path": "../cached_data_used/kernels",
66
"bruteforced_caches_path": "../cached_data_used/cachefiles",
77
"visualization_caches_path": "../cached_data_used/visualizations",
88
"kernels": [

‎experiment_files/test_scatter.json‎

Lines changed: 55 additions & 55 deletions
Original file line numberDiff line numberDiff line change
@@ -1,59 +1,59 @@
11
{
2-
"version": "0.1.2",
3-
"name": "Test random search (calculated vs executed)",
4-
"folder_id": "test_random_calculated",
5-
"kernels": [
6-
"convolution"
7-
],
8-
"kernel_path": "cached_data_used/kernels",
9-
"GPUs": [
10-
"RTX_2080_Ti"
11-
],
12-
"minimization": true,
13-
"resolution": 1e3,
14-
"cutoff_percentile": 0.95,
15-
"cutoff_percentile_start": 0.1,
16-
"objective_time_keys": [
17-
"strategy_time",
18-
"compile_time",
19-
"benchmark_time",
20-
"verification_time",
21-
"framework_time"
22-
],
23-
"objective_performance_keys": [
24-
"time"
25-
],
26-
"cutoff_type": "fevals",
27-
"plot": {
28-
"plot_x_value_types": [
29-
"fevals",
30-
"time"
2+
"version": "0.1.2",
3+
"name": "Test random search (calculated vs executed)",
4+
"folder_id": "test_random_calculated",
5+
"kernels": [
6+
"convolution"
317
],
32-
"plot_y_value_types": [
33-
"absolute"
8+
"kernels_path": "cached_data_used/kernels",
9+
"GPUs": [
10+
"RTX_2080_Ti"
3411
],
35-
"confidence_level": 0.95
36-
},
37-
"strategy_defaults": {
38-
"repeats": 100,
39-
"minimum_number_of_evaluations": 20,
40-
"stochastic": true,
41-
"record_data": [
42-
"time",
43-
"GFLOP/s"
44-
]
45-
},
46-
"strategies": [
47-
{
48-
"name": "random_sample",
49-
"strategy": "random_sample",
50-
"display_name": "Random sampling 500 iters",
51-
"repeats": 500
12+
"minimization": true,
13+
"resolution": 1e3,
14+
"cutoff_percentile": 0.95,
15+
"cutoff_percentile_start": 0.1,
16+
"objective_time_keys": [
17+
"strategy_time",
18+
"compile_time",
19+
"benchmark_time",
20+
"verification_time",
21+
"framework_time"
22+
],
23+
"objective_performance_keys": [
24+
"time"
25+
],
26+
"cutoff_type": "fevals",
27+
"plot": {
28+
"plot_x_value_types": [
29+
"fevals",
30+
"time"
31+
],
32+
"plot_y_value_types": [
33+
"absolute"
34+
],
35+
"confidence_level": 0.95
5236
},
53-
{
54-
"name": "genetic_algorithm",
55-
"strategy": "genetic_algorithm",
56-
"display_name": "Genetic Algorithm"
57-
}
58-
]
59-
}
37+
"strategy_defaults": {
38+
"repeats": 100,
39+
"minimum_number_of_evaluations": 20,
40+
"stochastic": true,
41+
"record_data": [
42+
"time",
43+
"GFLOP/s"
44+
]
45+
},
46+
"strategies": [
47+
{
48+
"name": "random_sample",
49+
"strategy": "random_sample",
50+
"display_name": "Random sampling 500 iters",
51+
"repeats": 500
52+
},
53+
{
54+
"name": "genetic_algorithm",
55+
"strategy": "genetic_algorithm",
56+
"display_name": "Genetic Algorithm"
57+
}
58+
]
59+
}

‎src/autotuning_methodology/experiments.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -142,10 +142,10 @@ def execute_experiment(filepath: str, profiling: bool = False) -> tuple[dict, di
142142
strategies = get_strategies(experiment)
143143

144144
# add the kernel directory to the path to import the module, relative to the experiment file
145-
kernel_path = experiment_folderpath / Path(experiment["kernel_path"])
146-
if not kernel_path.exists():
147-
raise FileNotFoundError(f"No such path {kernel_path}, CWD: {getcwd()}")
148-
sys.path.append(str(kernel_path))
145+
kernels_path = experiment_folderpath / Path(experiment["kernels_path"])
146+
if not kernels_path.exists():
147+
raise FileNotFoundError(f"No such path {kernels_path}, CWD: {getcwd()}")
148+
sys.path.append(str(kernels_path))
149149
kernel_names = experiment["kernels"]
150150
kernels = list(import_module(kernel_name) for kernel_name in kernel_names)
151151

‎src/autotuning_methodology/schema.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,18 @@
1818
"description": "Unique ID of the folder to store the results of this experiment in",
1919
"type": "string"
2020
},
21+
"kernels_path": {
22+
"description": "Path to the directory that has the tuning scripts specified in `kernels`, relative to the experiments file.",
23+
"type": "string"
24+
},
25+
"bruteforced_caches_path": {
26+
"description": "Path to the directory that has the bruteforced caches, relative to the experiments file.",
27+
"type": "string"
28+
},
29+
"visualization_caches_path": {
30+
"description": "Path to the directory to write / look for visualization caches, relative to the experiments file.",
31+
"type": "string"
32+
},
2133
"kernels": {
2234
"description": "Kernels to optimize",
2335
"type": "array",
@@ -126,6 +138,9 @@
126138
"version",
127139
"name",
128140
"folder_id",
141+
"kernels_path",
142+
"bruteforced_caches_path",
143+
"visualization_caches_path",
129144
"kernels",
130145
"GPUs",
131146
"minimization",

‎tests/autotuning_methodology/integration/mockfiles/test.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,12 @@
22
"version": "0.1.2",
33
"name": "Mock run test",
44
"folder_id": "test_run_experiment",
5+
"kernels_path": "../mockfiles",
6+
"bruteforced_caches_path": "../../../../cached_data_used/cachefiles",
7+
"visualization_caches_path": "../../../../cached_data_used/visualizations",
58
"kernels": [
69
"mocktest_kernel_convolution"
710
],
8-
"kernel_path": "../mockfiles",
9-
"bruteforced_caches_path": "../../../../cached_data_used/cachefiles",
10-
"visualization_caches_path": "../../../../cached_data_used/visualizations",
1111
"GPUs": [
1212
"mock_GPU"
1313
],

‎tests/autotuning_methodology/integration/mockfiles/test_bad_kernel_path.json‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,12 @@
22
"version": "0.1.2",
33
"name": "Test output file writer",
44
"folder_id": "test_output_file_writer",
5+
"kernels_path": "cached_data_used/bogus_kernels_path",
6+
"bruteforced_caches_path": "../../../../cached_data_used/cachefiles",
7+
"visualization_caches_path": "../../../../cached_data_used/visualizations",
58
"kernels": [
69
"convolution"
710
],
8-
"kernel_path": "cached_data_used/bogus_kernel_path",
911
"GPUs": [
1012
"RTX_2080_Ti"
1113
],

0 commit comments

Comments
 (0)