Skip to content

Commit ec6151d

Browse files
japolsjapols
committed
Merge branch 'main' into feat/act_checkpoint_config
1 parent 3b40dba commit ec6151d

File tree

406 files changed

+25600
-9577
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

406 files changed

+25600
-9577
lines changed

‎.github/CODEOWNERS

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Workflows
2+
/.github/ @ecmwf/AnemoiSecurity
3+
4+
# Project configs
5+
/graphs/pyproject.toml @ecmwf/AnemoiSecurity
6+
/models/pyproject.toml @ecmwf/AnemoiSecurity
7+
/training/pyproject.toml @ecmwf/AnemoiSecurity
8+
9+
/.pre-commit-config.yaml @ecmwf/AnemoiSecurity
10+
/.release-please-config.json @ecmwf/AnemoiSecurity

‎.github/pull_request_template.md

Lines changed: 9 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -1,52 +1,15 @@
11
## Description
2+
<!-- What issue or task does this change relate to? -->
23

3-
<!-- Provide a brief summary of the changes introduced in this pull request. -->
4+
## What problem does this change solve?
5+
<!-- Describe if it's a bugfix, new feature, doc update, or breaking change -->
46

5-
## Type of Change
7+
## What issue or task does this change relate to?
8+
<!-- link to Issue Number -->
69

7-
- [ ] Bug fix (non-breaking change which fixes an issue)
8-
- [ ] New feature (non-breaking change which adds functionality)
9-
- [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
10-
- [ ] Documentation update
11-
12-
## Issue Number
13-
14-
<!-- Link the Issue number this change addresses, ideally in one of the "magic format" such as Closes #XYZ -->
15-
16-
<!-- Alternatively, explain the motivation behind the changes and the context in which they are being made. -->
17-
18-
## Code Compatibility
19-
20-
- [ ] I have performed a self-review of my code
21-
22-
### Code Performance and Testing
23-
24-
- [ ] I have added tests that prove my fix is effective or that my feature works
25-
- [ ] I ran the [complete Pytest test](https://anemoi.readthedocs.io/projects/training/en/latest/dev/testing.html) suite locally, and they pass
26-
- [ ] I have tested the changes on a single GPU
27-
- [ ] I have tested the changes on multiple GPUs / multi-node setups
28-
- [ ] I have run the Benchmark Profiler against the old version of the code
29-
- [ ] If the new feature introduces modifications at the config level, I have made sure to update Pydantic Schemas and default configs accordingly
30-
31-
32-
<!-- In case this affects the model sharding or other specific components please describe these here. -->
33-
34-
### Dependencies
35-
36-
- [ ] I have ensured that the code is still pip-installable after the changes and runs
37-
- [ ] I have tested that new dependencies themselves are pip-installable.
38-
- [ ] I have not introduced new dependencies in the inference portion of the pipeline
39-
40-
<!-- List any new dependencies that are required for this change and the justification to add them. -->
41-
42-
### Documentation
43-
44-
- [ ] My code follows the style guidelines of this project
45-
- [ ] I have updated the documentation and docstrings to reflect the changes
46-
- [ ] I have added comments to my code, particularly in hard-to-understand areas
47-
48-
<!-- Describe any major updates to the documentation -->
10+
## Additional notes ##
11+
<!-- Include any additional information, caveats, or considerations that the reviewer should be aware of. -->
4912

50-
## Additional Notes
13+
***As a contributor to the Anemoi framework, please ensure that your changes include unit tests, updates to any affected dependencies and documentation, and have been tested in a parallel setting (i.e., with multiple GPUs). As a reviewer, you are also responsible for verifying these aspects and requesting changes if they are not adequately addressed. For guidelines about those please refer to https://anemoi.readthedocs.io/en/latest/***
5114

52-
<!-- Include any additional information, caveats, or considerations that the reviewer should be aware of. -->
15+
By opening this pull request, I affirm that all authors agree to the [Contributor License Agreement.](https://github.com/ecmwf/codex/blob/main/Legal/contributor_license_agreement.md)

‎.github/workflows/integration-tests-hpc.yml

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
name: nightly-integration-tests-hpc-gpu
2+
3+
# Controls when the workflow will run
4+
on:
5+
schedule:
6+
- cron: "0 22 * * *" # every day at 22pm on default(main) branch
7+
workflow_dispatch:
8+
9+
10+
jobs:
11+
check_date:
12+
runs-on: ubuntu-latest
13+
name: Check latest commit
14+
outputs:
15+
should_run: ${{ steps.should_run.outputs.should_run }}
16+
steps:
17+
- uses: actions/checkout@v4
18+
- name: print latest_commit
19+
run: echo ${{ github.sha }}
20+
- id: should_run
21+
continue-on-error: true
22+
name: check latest commit is less than a day
23+
if: ${{ github.event_name == 'schedule' }}
24+
run: test -z $(git rev-list --after="24 hours" ${{ github.sha }}) && echo "::set-output name=should_run::false"
25+
integration-tests:
26+
runs-on: hpc
27+
needs: check_date
28+
if: ${{ needs.check_date.outputs.should_run != 'false' }}
29+
steps:
30+
- uses: ecmwf-actions/reusable-workflows/ci-hpc-generic@v2
31+
with:
32+
template: |
33+
set -eux
34+
module load python3/3.12.9-01
35+
REPO_NAME=anemoi-core
36+
REPO_BRANCH=${{ github.head_ref || github.ref_name }}
37+
cd $TMPDIR
38+
git clone -b $REPO_BRANCH https://${{ secrets.GH_REPO_READ_TOKEN }}@github.com/ecmwf/${REPO_NAME}.git
39+
cd $REPO_NAME
40+
python3 -m venv .venv
41+
source .venv/bin/activate
42+
pip install --upgrade pip
43+
pip install -e ./training[all,tests] -e ./models[all,tests] -e ./graphs[all,tests]
44+
python3 -m pytest -v training/tests/integration --slow
45+
deactivate
46+
rm -rf $REPO_NAME
47+
sbatch_options: |
48+
#SBATCH --job-name=anemoi_core_pytest
49+
#SBATCH --time=01:00:00
50+
#SBATCH --qos=ng
51+
#SBATCH --gpus=1
52+
#SBATCH --gres=gpu:1
53+
#SBATCH --mem=16G
54+
troika_user: ${{ secrets.HPC_TEST_USER }}
55+
56+
notify-team:
57+
needs: integration-tests
58+
if: failure() && github.event_name == 'schedule'
59+
runs-on: ubuntu-latest
60+
steps:
61+
- name: Add failure comment
62+
uses: peter-evans/create-or-update-comment@v3
63+
with:
64+
issue-number: 294
65+
body: |
66+
## 🚨 Nightly Job Failed
67+
68+
**Repository:** ${{ github.repository }}
69+
**Workflow:** ${{ github.workflow }}
70+
**Run URL:** [View logs](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
71+
**Time:** ${{ github.event.schedule }}
72+
73+
cc: @ecmwf/anemoisecurity

‎.github/workflows/integration-tests.yml

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,5 +15,24 @@ jobs:
1515
python-version: '3.10'
1616
install-package: "-e ./training[all,tests] -e ./graphs[all,tests] -e ./models[all,tests]"
1717
test-directory: "training/tests/integration"
18-
pytest-options: "--longtests"
18+
pytest-options: "--slow"
1919
manual: ${{github.event_name == 'workflow_dispatch' && 'true' || 'false'}}
20+
21+
notify-team:
22+
needs: integration_tests
23+
if: failure() && github.event_name == 'schedule'
24+
runs-on: ubuntu-latest
25+
steps:
26+
- name: Add failure comment
27+
uses: peter-evans/create-or-update-comment@v3
28+
with:
29+
issue-number: 294
30+
body: |
31+
## 🚨 Nightly Job Failed
32+
33+
**Repository:** ${{ github.repository }}
34+
**Workflow:** ${{ github.workflow }}
35+
**Run URL:** [View logs](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})
36+
**Time:** ${{ github.event.schedule }}
37+
38+
cc: @ecmwf/anemoisecurity

‎.github/workflows/label-public-pr.yml

Lines changed: 0 additions & 10 deletions
This file was deleted.

‎.github/workflows/pr-conventional-commit.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This workflow ensures that the PR title follows the Conventional Commit format.
2-
name: "[Pull Request] Ensure Conventional Commit in PR title"
2+
name: "[PR] Ensure Conventional Commit Title"
33

44
on:
55
pull_request_target:

‎.github/workflows/pr-label-ats.yml

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# This workflow checks that the appropriate ATS labels are applied to a pull request:
2+
# - "ATS Approval Not Needed": pass
3+
# - "ATS Approved": pass
4+
# - "ATS Approval Needed": fail
5+
# - Missing ATS label: fail
6+
7+
name: "[PR] ATS labels"
8+
9+
on:
10+
pull_request:
11+
types:
12+
- opened
13+
- edited
14+
- reopened
15+
- labeled
16+
- unlabeled
17+
- synchronize
18+
19+
permissions:
20+
pull-requests: read
21+
22+
jobs:
23+
check:
24+
runs-on: ubuntu-latest
25+
steps:
26+
- name: Get PR details
27+
uses: actions/github-script@v7
28+
id: check-pr
29+
with:
30+
script: |
31+
const pr = await github.rest.pulls.get({
32+
owner: context.repo.owner,
33+
repo: context.repo.repo,
34+
pull_number: context.payload.pull_request.number,
35+
});
36+
const labels = pr.data.labels.map(label => label.name);
37+
core.setOutput('labels', JSON.stringify(labels));
38+
core.setOutput('author', context.payload.pull_request.user.login);
39+
40+
- name: Evaluate ATS labels
41+
run: |
42+
AUTHOR='${{ steps.check-pr.outputs.author }}'
43+
if [ "$AUTHOR" == "DeployDuck" ] || [ "$AUTHOR" == "pre-commit-ci[bot]" ]; then
44+
echo "Bot PR, skipping."
45+
exit 0
46+
fi
47+
LABELS=$(echo '${{ steps.check-pr.outputs.labels }}' | jq -r '.[]')
48+
echo "Labels found:"
49+
echo -e "$LABELS\n"
50+
echo "Result:"
51+
if echo "$LABELS" | grep -qi "ATS approval not needed"; then
52+
echo "ATS approval not needed. Passing."
53+
EXIT_CODE=0
54+
elif echo "$LABELS" | grep -qi "ATS approved"; then
55+
echo "ATS Approved. Passing."
56+
EXIT_CODE=0
57+
elif echo "$LABELS" | grep -qi "ATS approval needed"; then
58+
echo "ATS Approval Needed. Failing."
59+
EXIT_CODE=1
60+
else
61+
echo "No ATS approval labels found. Please assign the appropriate ATS label. Failing."
62+
EXIT_CODE=1
63+
fi
64+
echo -e "\nFor more information on ATS labels, see:"
65+
echo "https://anemoi.readthedocs.io/en/latest/contributing/guidelines.html#labelling-guidelines"
66+
exit $EXIT_CODE

‎.github/workflows/pr-label-conventional-commits.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# This workflow assigns labels to a pull request based on the Conventional Commits format.
22
# This is necessary for release-please to work properly.
3-
name: "[Pull Request] Label Conventional Commits"
3+
name: "[PR] Label Conventional Commits"
44

55
on:
66
pull_request:

‎.github/workflows/pr-label-file-based.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# This workflow assigns labels to a pull request based on the files changed in the PR.
22
# The labels are defined in the `.github/labels.yml` file.
3-
name: "[Pull Request] Label File-based"
3+
name: "[PR] Label File-based"
44
on:
55
pull_request_target:
66
types: [opened, synchronize]

‎.github/workflows/pr-label-public.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Manage labels of pull requests that originate from forks
2-
name: "[Pull Request] Label PRs from public forks"
2+
name: "[PR] Label Forks"
33

44
on:
55
pull_request_target:

0 commit comments

Comments
 (0)