Skip to content

Commit 1f7afd1

Browse files
valentin-seitzValentin SeitzpreCICE Tests VMMakisH
authored
Add Workflows for generating reference results (#410)
* add machine info to reference results * rename script to unify naming * add workflows * make autopep happy and trigger wf once :) * made wf callable * remove push trigger * added some switch case to check for the different names until changes propagate * Fix non terminating timeouts (#421) Co-authored-by: Valentin Seitz <valentin.seitz@tum.de> * Systests/move to commits (#422) * remove non working gha cache stuff * Move to only commits in the dockerfile --------- Co-authored-by: Valentin Seitz <valentin.seitz@tum.de> * adopt new naming schema * filter lscpu to hide vulnerabilities * move to consistent naming schema * Adding reference results from precice-tests vm * Make autopep8 * make paths a bit nicer and easier to understand * Update tools/tests/README.md Co-authored-by: Gerasimos Chourdakis <chourdak@in.tum.de> * Update tools/tests/README.md Co-authored-by: Gerasimos Chourdakis <chourdak@in.tum.de> * introduce variable instead of magic number 10 * added comment on very long rm instructions --------- Co-authored-by: Valentin Seitz <valentin.seitz@tum.de> Co-authored-by: preCICE Tests VM <tests@precice.org> Co-authored-by: Gerasimos Chourdakis <chourdak@in.tum.de>
1 parent 622caf7 commit 1f7afd1

23 files changed

+277
-75
lines changed

‎.github/workflows/generate_reference_results_manual.yml

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
name: Generate reference results (manual)
2+
on:
3+
workflow_dispatch:
4+
inputs:
5+
from_ref:
6+
description: 'Use the systemtests + tutorial metadata + reference_version from this ref'
7+
required: true
8+
type: string
9+
commit_msg:
10+
description: 'Commit msg for commit that adds the reference results'
11+
default: "Adding reference results"
12+
type: string
13+
loglevel:
14+
description: 'loglevel used for the systemtests'
15+
default: 'INFO'
16+
required: true
17+
type: choice
18+
options:
19+
- 'DEBUG'
20+
- 'INFO'
21+
- 'WARNING'
22+
- 'ERROR'
23+
- 'CRITICAL'
24+
25+
jobs:
26+
generate_reference_results_manual:
27+
uses: ./.github/workflows/generate_reference_results_workflow.yml
28+
with:
29+
from_ref: ${{ inputs.from_ref }}
30+
commit_msg: ${{ inputs.commit_msg }}
31+
loglevel: ${{ inputs.loglevel }}

‎.github/workflows/generate_reference_results_workflow.yml

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: Generate reference results workflow
2+
on:
3+
workflow_call:
4+
inputs:
5+
from_ref:
6+
description: 'Use the systemtests + tutorial metadata + reference_version from this ref'
7+
required: true
8+
type: string
9+
commit_msg:
10+
description: 'Commit msg for commit that adds the reference results'
11+
default: "Adding reference results"
12+
type: string
13+
loglevel:
14+
description: 'loglevel used for the systemtests'
15+
default: 'INFO'
16+
required: true
17+
type: string
18+
jobs:
19+
generate_reference_results:
20+
runs-on: [self-hosted, linux, x64, precice-tests-vm]
21+
steps:
22+
- name: Display a quick job summary
23+
run: |
24+
echo "Initiated by: ${{ github.actor }}"
25+
echo "Running generate_reference_results.py --log-level ${{inputs.loglevel}}"
26+
echo "Using Ref: ${{ inputs.from_ref }}"
27+
echo "Commit message on success: ${{ inputs.commit_msg }}"
28+
- name: Move LFS URL to local LFS server
29+
run: |
30+
/home/precice/runners_root/scripts/make_lfs_local.sh
31+
- name: 'Cleanup the folder'
32+
# The first rf -rf ./* removes all non hidden files
33+
# The second rf -rf ./.??* removes all hidden files (but not . and ..)
34+
run: |
35+
ls -la ./
36+
rm -rf ./*
37+
rm -rf ./.??*
38+
ls -la ./
39+
- name: Check out Tutorials for systest
40+
uses: actions/checkout@v4
41+
with:
42+
ref: ${{ inputs.from_ref }}
43+
lfs: true
44+
fetch-depth: 0
45+
- name: Install Python dependencies
46+
run: |
47+
pip install --user -r tools/tests/requirements.txt
48+
- name: Run tests
49+
run: |
50+
cd tools/tests
51+
test -f generate_reference_results.py && export GENERATE_REF_RESULTS=generate_reference_results.py
52+
test -f generate_reference_data.py && export GENERATE_REF_RESULTS=generate_reference_data.py
53+
echo "Selected $GENERATE_REF_RESULTS to run"
54+
python $GENERATE_REF_RESULTS --log-level=${{inputs.loglevel}}
55+
cd ../../
56+
- name: Create commit
57+
if: success()
58+
run: |
59+
git checkout ${{ inputs.from_ref }}
60+
git add ./*/*/*.tar.gz
61+
git add ./*/*.tar.gz
62+
git add ./*/*.metadata
63+
git commit -m "${{inputs.commit_msg}}"
64+
git push
65+
- name: Upload artifacts for debugging
66+
if: failure()
67+
uses: actions/upload-artifact@v3
68+
with:
69+
name: runs
70+
path: |
71+
runs/*

‎flow-over-heated-plate/reference-data/fluid-openfoam_solid-fenics.tar.gz

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎flow-over-heated-plate/reference-data/fluid-openfoam_solid-nutils.tar.gz

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎flow-over-heated-plate/reference-data/fluid-openfoam_solid-openfoam.tar.gz

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎flow-over-heated-plate/reference-results/fluid-openfoam_solid-fenics.tar.gz

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:65d4c461e3f6167846730de14b0778b004a02f2db740f2c263aee4e2bd8b86a6
3+
size 778312

‎flow-over-heated-plate/reference-results/fluid-openfoam_solid-nutils.tar.gz

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:7d790bf8a032b1f155699b5a63287124be202d45bcb59cab9b320c3fef86ba56
3+
size 533210

‎flow-over-heated-plate/reference-results/fluid-openfoam_solid-openfoam.tar.gz

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:2ab9f3f762beec987e0a90ce99e5e3b85f406e4738599fd4b1ad68942ed7f516
3+
size 498933

‎flow-over-heated-plate/reference_results.metadata

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,15 @@ This File has been generated by the generate_reference_results.py and should not
55
# Reference Results
66

77
This file contains an overview of the results over the reference results as well as the arguments used to generate them.
8+
We also include some information on the machine used to generate them
89

910
## List of files
1011

1112
| name | time | sha256 |
1213
|------|------|-------|
13-
| fluid-openfoam_solid-nutils.tar.gz | 2023-09-04 23:41:09 | 8b3157902d3ad78593c5471a3a94dd5559fe6970e4cb54658b22ffd270c00297 |
14-
| fluid-openfoam_solid-fenics.tar.gz | 2023-09-04 23:41:09 | 2565ffc51c8d80fab06ccdda33f62efceb6cdc03318998f5adc9ed0ac84acac2 |
15-
| fluid-openfoam_solid-openfoam.tar.gz | 2023-09-04 23:41:09 | c29989e0118ade759e7313330aa8d3d900c660c404eaacd529e979aae6b899c7 |
14+
| fluid-openfoam_solid-nutils.tar.gz | 2023-12-13 22:22:18 | 7d790bf8a032b1f155699b5a63287124be202d45bcb59cab9b320c3fef86ba56 |
15+
| fluid-openfoam_solid-fenics.tar.gz | 2023-12-13 22:22:18 | 65d4c461e3f6167846730de14b0778b004a02f2db740f2c263aee4e2bd8b86a6 |
16+
| fluid-openfoam_solid-openfoam.tar.gz | 2023-12-13 22:22:18 | 2ab9f3f762beec987e0a90ce99e5e3b85f406e4738599fd4b1ad68942ed7f516 |
1617

1718
## List of arguments used to generate the files
1819

@@ -27,3 +28,36 @@ This file contains an overview of the results over the reference results as well
2728
| PLATFORM | ubuntu_2204 |
2829
| CALULIX_VERSION | 2.20 |
2930
| CALULIX_ADAPTER_REF | v2.20.0 |
31+
## Information about the machine
32+
33+
### uname -a
34+
35+
Linux precice-tests 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
36+
37+
38+
### lscpu
39+
40+
Architecture: x86_64
41+
CPU op-mode(s): 32-bit, 64-bit
42+
Address sizes: 45 bits physical, 48 bits virtual
43+
Byte Order: Little Endian
44+
CPU(s): 4
45+
On-line CPU(s) list: 0-3
46+
Vendor ID: GenuineIntel
47+
Model name: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
48+
CPU family: 6
49+
Model: 85
50+
Thread(s) per core: 1
51+
Core(s) per socket: 1
52+
Socket(s): 4
53+
Stepping: 4
54+
BogoMIPS: 4199.99
55+
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat pku ospke md_clear flush_l1d arch_capabilities
56+
Hypervisor vendor: VMware
57+
Virtualization type: full
58+
L1d cache: 128 KiB (4 instances)
59+
L1i cache: 128 KiB (4 instances)
60+
L2 cache: 4 MiB (4 instances)
61+
L3 cache: 88 MiB (4 instances)
62+
NUMA node(s): 1
63+
NUMA node0 CPU(s): 0-3

‎perpendicular-flap/reference-data/fluid-openfoam_solid-calculix.tar.gz

Lines changed: 0 additions & 3 deletions
This file was deleted.

‎perpendicular-flap/reference-results/fluid-openfoam_solid-calculix.tar.gz

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
version https://git-lfs.github.com/spec/v1
2+
oid sha256:905c90f4f5c7776c5822b9235e435ae25b9097d2f8bc6392865402b21a525194
3+
size 13559914

‎perpendicular-flap/reference_results.metadata

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,13 @@ This File has been generated by the generate_reference_results.py and should not
55
# Reference Results
66

77
This file contains an overview of the results over the reference results as well as the arguments used to generate them.
8+
We also include some information on the machine used to generate them
89

910
## List of files
1011

1112
| name | time | sha256 |
1213
|------|------|-------|
13-
| fluid-openfoam_solid-calculix.tar.gz | 2023-09-04 23:41:09 | 596fe1aec3f72fa194c37c485688dea4e308544e3cd773d7c815d6e27c4e55a8 |
14+
| fluid-openfoam_solid-calculix.tar.gz | 2023-12-13 22:22:18 | 905c90f4f5c7776c5822b9235e435ae25b9097d2f8bc6392865402b21a525194 |
1415

1516
## List of arguments used to generate the files
1617

@@ -25,3 +26,36 @@ This file contains an overview of the results over the reference results as well
2526
| PLATFORM | ubuntu_2204 |
2627
| CALULIX_VERSION | 2.20 |
2728
| CALULIX_ADAPTER_REF | v2.20.0 |
29+
## Information about the machine
30+
31+
### uname -a
32+
33+
Linux precice-tests 5.15.0-89-generic #99-Ubuntu SMP Mon Oct 30 20:42:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
34+
35+
36+
### lscpu
37+
38+
Architecture: x86_64
39+
CPU op-mode(s): 32-bit, 64-bit
40+
Address sizes: 45 bits physical, 48 bits virtual
41+
Byte Order: Little Endian
42+
CPU(s): 4
43+
On-line CPU(s) list: 0-3
44+
Vendor ID: GenuineIntel
45+
Model name: Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
46+
CPU family: 6
47+
Model: 85
48+
Thread(s) per core: 1
49+
Core(s) per socket: 1
50+
Socket(s): 4
51+
Stepping: 4
52+
BogoMIPS: 4199.99
53+
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat pku ospke md_clear flush_l1d arch_capabilities
54+
Hypervisor vendor: VMware
55+
Virtualization type: full
56+
L1d cache: 128 KiB (4 instances)
57+
L1i cache: 128 KiB (4 instances)
58+
L2 cache: 4 MiB (4 instances)
59+
L3 cache: 88 MiB (4 instances)
60+
NUMA node(s): 1
61+
NUMA node0 CPU(s): 0-3

‎tools/tests/README.md

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -38,17 +38,17 @@ gh workflow run run_testsuite_manual.yml -f suites=fenics_test --ref=develop
3838
Another example, to use the latest develop branches and enable debug information of the tests:
3939

4040
```shell
41-
gh workflow run run_testsuite_manual.yml -f suites=fenics_test -f build_args="PRECICE_REF:develop,OPENFOAM_ADAPTER_REF:develop,PYTHON_BINDINGS_REF:develop,FENICS_ADAPTER_REF:develop" -f loglevel=DEBUG --ref=develop
41+
gh workflow run run_testsuite_manual.yml -f suites=fenics_test -f build_args="PRECICE_REF:v2.5.0,OPENFOAM_ADAPTER_REF:v1.2.3,PYTHON_BINDINGS_REF:v2.5.0.4,FENICS_ADAPTER_REF:v1.4.0" -f loglevel=DEBUG --ref=develop
4242
```
4343

44-
where the `*_REF` can also be specific Git commits.
44+
where the `*_REF` should be a specific [commit-ish](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddefcommit-ishacommit-ishalsocommittish).
4545

4646
Example output:
4747

4848
```text
4949
Run cd tools/tests
5050
cd tools/tests
51-
python systemtests.py --build_args=PRECICE_REF:develop,OPENFOAM_ADAPTER_REF:develop,PYTHON_BINDINGS_REF:develop,FENICS_ADAPTER_REF:develop --suites=fenics_test --log-level=DEBUG
51+
python systemtests.py --build_args=PRECICE_REF:v2.5.0,OPENFOAM_ADAPTER_REF:v1.2.3,PYTHON_BINDINGS_REF:v2.5.0.4,FENICS_ADAPTER_REF:v1.4.0 --suites=fenics_test --log-level=DEBUG
5252
cd ../../
5353
shell: /usr/bin/bash -e {0}
5454
INFO: About to run the following systemtest in the directory /home/precice/runners_root/actions-runner-tutorial/_work/tutorials/tutorials/runs:
@@ -76,18 +76,9 @@ In this case, building and running seems to work out, but the tests fail because
7676

7777
## Understanding what went wrong
7878

79-
Let's first see how the workflow was triggered. If we expand the `Set up job` step, we can see the inputs provided:
80-
81-
```text
82-
Inputs
83-
build_args: PRECICE_REF:develop,OPENFOAM_ADAPTER_REF:develop,PYTHON_BINDINGS_REF:develop,FENICS_ADAPTER_REF:develop
84-
loglevel: DEBUG
85-
suites: fenics_test
86-
systests_branch: develop
87-
upload_artifacts: FALSE
88-
```
89-
90-
In the summary, we can find the results and more logs as a build artifact. This includes two interesting files: `stdout.log` and `stderr.log`. These include all Docker build steps and the simulation output, as well as the exact git clone command.
79+
The easiest way to debug a systemtest run is first to have a look at the output written into the action on GitHub.
80+
If this does not provide enough hints, the next step is to download the generated `runs` artifact. Note that by default this will only be generated if the systemtests fail.
81+
Inside the archive, a test-specific subfolder like `flow-over-heated-plate_fluid-openfoam-solid-fenics_2023-11-19-211723` contains two log files: a `stderr.log` and `stdout.log`. This can be a starting point for a further investigation.
9182

9283
## Adding new tests
9384

@@ -138,7 +129,7 @@ Metadata and workflow/script files:
138129
- ...
139130
- `metadata.yml`: describes each case directory (which participant, which component, which script to run, ...)
140131
- `tools/tests/`
141-
- `component-templates/`: jinja2 templates for Docker Compose services, specifying cache system
132+
- `component-templates/`: jinja2 templates for Docker Compose services for the components
142133
- `calculix-adapter.yaml`
143134
- `fenics-adapter.yaml`
144135
- `openfoam-adapter.yaml`
@@ -149,7 +140,7 @@ Metadata and workflow/script files:
149140
- `docker-compose.field_compare.template.yaml`: Describes how to compare results with fieldcompare (Docker Compose service template)
150141
- `components.yaml`: Declares the available components and their parameters/options
151142
- `reference_results.metadata.template`: Template for reporting the versions used to generate the reference results
152-
- `reference_versions.yaml`: Versions of components to use for generating the reference results
143+
- `reference_versions.yaml`: List of arguments to use for generating the reference results
153144
- `tests.yaml`: Declares the available tests, grouped in test suites
154145

155146
User-facing tools:
@@ -160,7 +151,7 @@ User-facing tools:
160151
- `print_metadata.py`: Prints the metadata of each tutorial that contains a `metadata.yaml` file.
161152
- `print_case_combinations.py`: Prints all possible combinations of tutorial cases, using the `metadata.yaml` files.
162153
- `build_docker_images.py`: Build the Docker images for each test
163-
- `generate_reference_data.py`: Executes the system tests with the versions defined in `reference_versions.yaml` and generates the reference data archives, with the names described in `tests.yaml`.
154+
- `generate_reference_results.py`: Executes the system tests with the versions defined in `reference_versions.yaml` and generates the reference data archives, with the names described in `tests.yaml`. (should only be used by the CI Pipeline)
164155

165156
Implementation scripts:
166157

@@ -226,7 +217,7 @@ The components mentioned in the Metadata are defined in the central `components.
226217
openfoam-adapter:
227218
repository: https://github.com/precice/openfoam-adapter
228219
template: component-templates/openfoam-adapter.yaml
229-
build_arguments: # these things mean something to the docker-service
220+
build_arguments:
230221
PRECICE_REF:
231222
description: Version of preCICE to use
232223
default: "main"
@@ -247,15 +238,15 @@ openfoam-adapter:
247238

248239
This `openfoam-adapter` component has the following attributes:
249240

250-
- `repository`: URL to the Git project (without the `.git` extension), used to fetch the component
241+
- `repository`: URL to the Git projects
251242
- `template`: A template for a Docker Compose service of this component
252243
- `build_arguments`: Arguments passed to the Docker Compose service (arbitrary)
253244

254245
#### Naming schema for build_arguments
255246

256247
Since the docker containers are still a bit mixed in terms of capabilities and support for different build_argument combinations the following rules apply:
257248

258-
- A build_argument ending in **_REF** means that it refers to a git reference (like a branch or commit) beeing used to build the image.
249+
- A build_argument ending in **_REF** means that it refers to a git commit-ish (like a tag or commit) beeing used to build the image. Its important to not use branch names here as we heavily rely on dockers build cache to speedup things. But since the input variable to the docker builder will not change, we might have wrong cache hits.
259250
- All other build_arguments are free of rules and up to the container maintainer.
260251

261252
### Component templates
@@ -313,8 +304,19 @@ This defines two test suites, namely `openfoam_adapter_pr` and `openfoam_adapter
313304

314305
### Generate Reference Results
315306

307+
#### via GitHub workflow (recommended)
308+
309+
The preferred way of adding reference results is via the manual triggerable `Generate reference results (manual)` workflow. This takes two inputs:
310+
311+
- `from_ref`: branch where the new test configuration (e.g added tests, new reference_versions.yaml) is
312+
- `commit_msg`: commit message for adding the reference results into the branch
313+
314+
The workflow will checkout the `from_ref`, take the status of the systemtests of that branch and execute `python generate_reference_results.py`, upload the LFS objects into the self-hosted LFS server and add a commit with `commit_msg` onto the `from_ref` branch.
315+
316+
#### manually
317+
316318
In order to generate the reference results edit the `reference_versions.yaml` to match the required `build_arguments` otherwise passed via the cli.
317-
Executing `generate_reference_data.py` will then generate a the following files:
319+
Executing `generate_reference_results.py` will then generate the following files:
318320

319321
- all distinct `.tar.gz` defined in the `tests.yaml`
320322
- a `reference_results.md` in the tutorial folder describing the arguments used and a sha-1 hash of the `tar.gz` archive.

‎tools/tests/component-templates/calculix-adapter.yaml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,6 @@ build:
55
- {{key}}={{value}}
66
{% endfor %}
77
target: calculix_adapter
8-
cache_from:
9-
- type=gha
10-
cache_to:
11-
- type=gha,mode=min,scope=calculix_adapter
128
depends_on:
139
prepare:
1410
condition: service_completed_successfully

‎tools/tests/component-templates/fenics-adapter.yaml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,6 @@ build:
55
- {{key}}={{value}}
66
{% endfor %}
77
target: fenics_adapter
8-
cache_from:
9-
- type=gha
10-
cache_to:
11-
- type=gha,mode=min,scope=fenics_adapter
128
depends_on:
139
prepare:
1410
condition: service_completed_successfully

‎tools/tests/component-templates/nutils-adapter.yaml

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,6 @@ build:
55
- {{key}}={{value}}
66
{% endfor %}
77
target: nutils_adapter
8-
cache_from:
9-
- type=gha
10-
cache_to:
11-
- type=gha,mode=min,scope=nutils_adapter
128
depends_on:
139
prepare:
1410
condition: service_completed_successfully

0 commit comments

Comments
 (0)