Skip to content

Bundle staging PRs #213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 218 commits into
base: main
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
218 commits
Select commit Hold shift + click to select a range
c6fb707
initial version of grouping tarballs in a single staging PR
truib Apr 22, 2025
aa32e71
add code linting check
truib Apr 22, 2025
88ba094
revise make_approval_request and fix flake8 issues
truib Apr 22, 2025
dd9b71a
requirements.txt is not needed
truib Apr 22, 2025
2cfde96
fix flake8 issues in automated_ingestion.py
truib Apr 22, 2025
0cb2622
fix flake8 issues in check-stratum-servers.py
truib Apr 22, 2025
7214d27
incremental updates through testing
truib Apr 25, 2025
946dd65
add missing functions
truib Apr 25, 2025
51894ec
mark tarballs in group as new initially
truib Apr 25, 2025
ea51a5e
always use group branch only
truib Apr 25, 2025
7db3dbe
add a bit debug output
truib Apr 25, 2025
9ef3535
improve logging
truib Apr 26, 2025
d860d1e
more improvements to logging
truib Apr 26, 2025
18cf44d
tweak logging
truib Apr 26, 2025
4a6fecc
more logging output for downloads and signature verification
truib Apr 26, 2025
e1b1ee6
add capability to run verify script in a container
truib Apr 27, 2025
56a9d68
improve branch naming and only add files to staged dir in main branch
truib Apr 27, 2025
935025f
don't stop after staging files
truib Apr 27, 2025
682ddbd
add more log info when moving metadata files
truib Apr 27, 2025
412cc46
need to handle first tarball differently
truib Apr 27, 2025
d0f0d26
move from staged to approval plus a little more logging
truib Apr 27, 2025
c2c456d
fix missing function
truib Apr 27, 2025
5cad5ed
improve get_metadata_path
truib Apr 27, 2025
d045aa5
enhance logging capabilities
truib May 1, 2025
f9fd34e
add more logging scopes, convert logging calls and improve code reada…
truib May 1, 2025
aab9f6a
show function entry/exit at info level
truib May 1, 2025
678f0d7
tweak func leave msg and add context info
truib May 1, 2025
7ecb9be
fix shown file components and illustrate call stack depth
truib May 1, 2025
c65d1f7
convert logging calls in automated_ingestion.py
truib May 1, 2025
0206968
introducing task-based deployments
truib May 2, 2025
108bec3
support providing extensions, e.g., . .meta.txt
truib May 2, 2025
4fce92a
model remote/local files, download them
truib May 2, 2025
bb6351a
fix imports
truib May 2, 2025
8e64cc9
add more details to function entry/leave logging
truib May 2, 2025
c7a0254
fix details to function entry/leave logging
truib May 2, 2025
a0c8d4d
show actual function for entry/leave logging
truib May 2, 2025
e958b49
print actual lines of entry or leave
truib May 2, 2025
59c722b
determine lineno when leaving a function
truib May 2, 2025
39f4934
may only show approx line when leaving a function
truib May 2, 2025
12ea5ca
always use own S3 bucket/client, code refactoring and improvements
truib May 2, 2025
fe6b29c
fix aws config key names
truib May 2, 2025
01f7367
fix section name for secrets
truib May 2, 2025
9e84165
optimize download and improve choosing log scope and level
truib May 2, 2025
550de88
fix logging logic and remove obsolete decorator
truib May 2, 2025
dd5fcd4
make sure to use full log format incl level
truib May 2, 2025
bc184d8
use fixed length levelname
truib May 2, 2025
6ed3bdc
use fixed length levelname everywhere
truib May 2, 2025
28558b5
improve handling of download errors and cleanup of partially download…
truib May 2, 2025
7cf615e
add task description and verification of signatures
truib May 2, 2025
aa548dc
fix import of EESSITaskDescription
truib May 2, 2025
3e8f8bd
add more log output for verification
truib May 2, 2025
e51200d
add scopes to log messages and avoid message duplicates
truib May 2, 2025
bc94537
next attempt to avoid message duplicates
truib May 2, 2025
f9aa559
temporarily disable standard loggers
truib May 2, 2025
0f0dfca
change handling of temporary log handler for scopes
truib May 4, 2025
d87b078
change default download mode and optimise fetching of etags
truib May 4, 2025
55aa5a1
download task object when necessary
truib May 4, 2025
6838435
import DownloadMode
truib May 4, 2025
7afd283
initial EESSITask and support for handling states for different actions
truib May 4, 2025
49738c5
action is a field in task
truib May 4, 2025
f265d79
determine metadata/task state from GH staging repo
truib May 11, 2025
3959780
import missing Tuple
truib May 11, 2025
c5e45a1
flake8 improvements
truib May 11, 2025
67e74f3
add func entry/exit logging to EESSITask
truib May 11, 2025
c284692
change logginscope name and log info in _find_state
truib May 11, 2025
fbfc1ab
add missing scope for task ops
truib May 11, 2025
28a4745
add a bit more logging
truib May 11, 2025
d2b275d
obtain repo/pr from task or link2pr OR raise ValueError
truib May 11, 2025
ad7cb11
add logging for obtaining repo/pr from metadata
truib May 11, 2025
11a23a0
populate source from metadata (link2pr) and use it in _find_state
truib May 11, 2025
52d7bd3
hand over repo and pr number as argument
truib May 11, 2025
d0be2eb
fix Github exception issues
truib May 11, 2025
37ea684
add logging when listing dir contents
truib May 11, 2025
f170a25
set branch to default if needed
truib May 11, 2025
a795710
handle file not found exception
truib May 11, 2025
c63208f
fix element naming
truib May 11, 2025
2c0673f
first step towards obtaining payload when handling new task
truib May 16, 2025
7bcf5eb
add a bit logging when creating payload instance
truib May 16, 2025
795da23
code formatting improvements
truib May 16, 2025
4f14824
build up path to store task file in staging repo
truib May 16, 2025
14762ef
add functions to determine repo name and pr number
truib May 16, 2025
ffae1cf
use functions to determine repo name and pr number
truib May 16, 2025
066ad16
add metadata / task file to GH staging repo
truib May 16, 2025
d671c82
add function to return raw contents of metadata / task file
truib May 16, 2025
14d62e5
include sequence number in path and various improvements
truib May 17, 2025
2357abc
fix issue with non-existing class element
truib May 17, 2025
370b6a6
add a bit more logging when obtaining value from metadata
truib May 17, 2025
6dcb675
show data types of task and source elements
truib May 17, 2025
057a953
fix logic to obtain value for key from task or source
truib May 17, 2025
167cff2
use basename of remote_file_path
truib May 17, 2025
d700826
use task file name for storing state file in GH staging repo
truib May 17, 2025
a9dad68
fix indentation
truib May 17, 2025
bc6423b
various improvement for determining the state of a task
truib May 17, 2025
092f32b
add functions to handle sequences of deployments
truib May 22, 2025
f44d809
revise states of a task
truib Jun 1, 2025
3f9279f
start revising determining state
truib Jun 1, 2025
d0739f8
fix syntax
truib Jun 1, 2025
a068b72
fix various flake8 issues
truib Jun 1, 2025
c4a60d2
fix typing issue
truib Jun 1, 2025
0121bed
fix typing issue, take 2
truib Jun 1, 2025
a1b81e0
print task to be processed
truib Jun 1, 2025
0801ee3
only check default branch
truib Jun 1, 2025
e41670a
add main processing loop
truib Jun 1, 2025
9b0fb9a
fix missing state attribute
truib Jun 1, 2025
7fb88d3
remove recursion from handle and return current state
truib Jun 1, 2025
9b3bd00
commented out some logging and obsolete processing
truib Jun 1, 2025
7768a91
first version of handler for undetermined task state
truib Jun 1, 2025
e3ffa6c
remove trailing / from target_dir name
truib Jun 1, 2025
0fa6afb
fix target directory structure for NEW_TASK and skip creating symlink
truib Jun 1, 2025
570babb
add creation of symlink
truib Jun 1, 2025
f1f813e
add error handling and log messages
truib Jun 1, 2025
fc67e43
add _safe_create_file and error handling
truib Jun 1, 2025
467781c
improve logging
truib Jun 1, 2025
09fb092
create new tree in try/except block
truib Jun 1, 2025
6c6d3dd
debug issue creating git tree
truib Jun 1, 2025
0318816
improve error handling and reporting
truib Jun 1, 2025
b842eba
use InputGitTreeElement instead of simple Dict
truib Jun 1, 2025
29fd410
use pointer file instead of symlink
truib Jun 6, 2025
e6680d6
add a couple of files with a single commit
truib Jun 7, 2025
b63699b
add new line to state file
truib Jun 7, 2025
3a1d975
determine task state from TaskState file
truib Jun 7, 2025
6f2c92f
fix check for path
truib Jun 7, 2025
57b9da6
add log output when determining state
truib Jun 7, 2025
c53729f
fix from_string
truib Jun 7, 2025
af6e1d8
return upper case state name
truib Jun 7, 2025
c76b68b
use lower state name to create handler name
truib Jun 7, 2025
ed75818
convert task state to str first
truib Jun 7, 2025
bc61294
complete handler for state NEW_TASK
truib Jun 7, 2025
2ac47b6
add TODO about checking validity of sequence number and corresponding…
truib Jun 7, 2025
62bd006
first part for handling task after payload got staged
truib Jun 7, 2025
16f0a30
use sha for creating branch + make variable less ambiguous
truib Jun 7, 2025
2b0a191
simplify determination of state and obtain it from feature branch if …
truib Jun 8, 2025
4d1db2e
small improvements to handler for new_task
truib Jun 8, 2025
7eed7de
only use specific branch for determining state, and default to main b…
truib Jun 8, 2025
58c7bdc
use different method to determine feature branch name
truib Jun 8, 2025
f4bf917
create first version of PR plus some related improvements
truib Jun 8, 2025
9fda61b
filter some PRs, just during development
truib Jun 8, 2025
4686e0c
revise state updates to reflect current and future states
truib Jun 8, 2025
d4d08cf
filter one more PR
truib Jun 8, 2025
2c73ee9
alternative method to update taskstate file
truib Jun 8, 2025
d5cd773
also filter PR 18
truib Jun 8, 2025
902bae2
alternative method to update taskstate file
truib Jun 8, 2025
3561669
another attempt to avoid merge conflict
truib Jun 8, 2025
61fb944
yet another attempt at avoiding merge conflict
truib Jun 8, 2025
2cae150
improve readbility of code
truib Jun 9, 2025
9e6d4fe
fix obtaining arch value
truib Jun 9, 2025
029d5b6
improve commit message when changing task state
truib Jun 9, 2025
b996f6b
first step towards adding another task to a deployment PR
truib Jun 9, 2025
a4c2d02
don't stop processing after first task
truib Jun 9, 2025
121130a
first step to create contents overview
truib Jun 9, 2025
64c4fe8
a little more log output
truib Jun 9, 2025
1061e1e
one level up from target_dir
truib Jun 9, 2025
ae0ad4b
show basic task summary
truib Jun 15, 2025
7b4d267
use existing method to check for existance of task summary file
truib Jun 15, 2025
a754921
init payload object in constructor for EESSITask given corresponding …
truib Jun 15, 2025
8a37ae1
add comparison of TaskState values
truib Jun 15, 2025
088fee9
init payload to None initially
truib Jun 15, 2025
047d271
create payload analysis
truib Jun 16, 2025
eb3c1b1
add function to obtain URL for remote file
truib Jun 16, 2025
fe41646
fix var name in template
truib Jun 16, 2025
e3e10e3
create task summary file in staging PRR”
truib Jun 16, 2025
aa24612
first step to create PR contents overview
truib Jun 16, 2025
6a4f62c
use name of directory
truib Jun 16, 2025
1d7b707
add PR dir component
truib Jun 16, 2025
5e4f107
note TODO
truib Jun 16, 2025
71f703c
bump sequence number to 1
truib Jun 16, 2025
02c01d0
tweak formatting of bundling PR content
truib Jun 16, 2025
e508148
bump sequence number to 2
truib Jun 16, 2025
5c54754
alternative for creating PR body, bumping seq to 3
truib Jun 16, 2025
53670df
decode file contents from GitHub and bump sequence number
truib Jun 16, 2025
f0fc09f
need to access .content
truib Jun 16, 2025
0660fa1
add function to return bucket URL
truib Jun 16, 2025
2652589
use method that returns bucket URL
truib Jun 16, 2025
7b7bb63
bump sequence number to 5
truib Jun 16, 2025
6eb7002
add remote_file_path to bucket_url and update pull request
truib Jun 16, 2025
bd09060
bump sequence number to 6
truib Jun 16, 2025
c2b8513
implement first version of handler for PULL_REQUEST state
truib Jun 17, 2025
04e3e70
use different function to determine feature branch
truib Jun 18, 2025
3a45f93
fixing print statements in handlers
truib Jun 18, 2025
569de25
add more logging to main loop
truib Jun 18, 2025
8ad5cee
implement handler for approved state
truib Jun 18, 2025
623b8c3
fix access to payload file name
truib Jun 18, 2025
1cd5ebd
add method _issue_exists
truib Jun 18, 2025
c30abfc
updates to ingestion incl state handling, error handling
truib Jun 18, 2025
d9d2fc8
fix using state and remove unused function transition_to
truib Jun 18, 2025
96050b6
enable code to run ingestion script
truib Jun 21, 2025
435b96f
bump sequence number to 7
truib Jun 21, 2025
37149ee
log result and output of ingest script
truib Jun 21, 2025
c7ed07d
bump sequence number to 8
truib Jun 21, 2025
d4528fe
bump sequence number to 9
truib Jun 21, 2025
73a77ee
add logging for issue creation
truib Jun 21, 2025
adcb918
bump sequence number to 10
truib Jun 21, 2025
98c0409
improve logging when processing ingestion failure
truib Jun 21, 2025
da9cc94
add traceback when catching exception
truib Jun 21, 2025
54062a3
convert Path to str
truib Jun 21, 2025
bb6021e
bump sequence number to 11
truib Jun 21, 2025
de00cee
1st step to make sequence numbers non-hardcoded
truib Jun 22, 2025
b8240dd
rename target_dir to pull_request_dir
truib Jun 22, 2025
345bbb7
remove function to create symlink
truib Jun 22, 2025
e800415
remove function to obtain branch name from sequence number
truib Jun 22, 2025
d717527
remove one use of _get_fixed_sequence_number
truib Jun 22, 2025
868b23b
improve ways to obtain pull request directory
truib Jun 22, 2025
dd0e8bb
clarify how to determine pull_request_dir
truib Jun 22, 2025
20a7ae8
remove need for determining sequence number in _create_task_summary
truib Jun 22, 2025
814cd50
add function to determine sequence number and use the function
truib Jun 22, 2025
cff761e
improve logging when determining task state
truib Jun 22, 2025
d85e017
add logging and consider 'target_dir' attr name
truib Jun 22, 2025
cf08115
add logging for determining sequence number and pull request
truib Jun 22, 2025
f94f69d
print head refs for all PRs
truib Jun 22, 2025
06faa4b
fix PR head ref logging
truib Jun 22, 2025
9979adf
do not use login when searching for PRs
truib Jun 22, 2025
af021ae
improve function to determine PR
truib Jun 22, 2025
251b60e
fix branch name
truib Jun 22, 2025
2275007
restructure logging when determining PRR”
truib Jun 22, 2025
cffc319
change way to determine PRs for branch name
truib Jun 22, 2025
29d17fc
little code cleanup
truib Jun 22, 2025
be9443c
reformat PR body and make format configurable
truib Jun 22, 2025
3713639
make config dictionary available for analyse_contents
truib Jun 22, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .flake8
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# This file is part of the EESSI filesystem layer,
# see https://github.com/EESSI/filesystem-layer
#
# author: Thomas Roeblitz (@trz42)
#
# license: GPLv2
#

[flake8]
max-line-length = 120

# ignore "Black would make changes" produced by flake8-black
# see also https://github.com/houndci/hound/issues/1769
extend-ignore = BLK100
36 changes: 36 additions & 0 deletions .github/workflows/check-flake8.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# This file is part of the EESSI filesystem layer,
# see https://github.com/EESSI/filesystem-layer
#
# author: Thomas Roeblitz (@trz42)
#
# license: GPLv2
#

name: Run tests
on: [push, pull_request]
# Declare default permissions as read only.
permissions: read-all
jobs:
test:
runs-on: ubuntu-22.04
strategy:
matrix:
python: [3.7, 3.8, 3.9, '3.10', '3.11', '3.12']
fail-fast: false
steps:
- name: checkout
uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0

- name: set up Python
uses: actions/setup-python@13ae5bb136fac2878aff31522b9efb785519f984 # v4.3.0
with:
python-version: ${{matrix.python}}

- name: Install required Python packages + pytest + flake8
run: |
python -m pip install --upgrade pip
python -m pip install --upgrade flake8

- name: Run flake8 to verify PEP8-compliance of Python code
run: |
flake8
48 changes: 48 additions & 0 deletions scripts/automated_ingestion/automated_ingestion.cfg.example
Original file line number Diff line number Diff line change
Expand Up @@ -63,13 +63,61 @@ pr_body = A new tarball has been staged for {pr_url}.
```

</details>

<details>
<summary>Overview of tarball contents</summary>

{tar_overview}

</details>

# Method for creating staging PRs:
# - 'individual': create one PR per tarball (old method)
# - 'grouped': group tarballs by link2pr and create one PR per group (new method)
staging_pr_method = individual

# Template for individual tarball PRs
individual_pr_body = A new tarball has been staged for {pr_url}.
Please review the contents of this tarball carefully.
Merging this PR will lead to automatic ingestion of the tarball to the repository {cvmfs_repo}.

<details>
<summary>Metadata of tarball</summary>

```
{metadata}
```

</details>

<details>
<summary>Overview of tarball contents</summary>

{tar_overview}

</details>

# Template for grouped tarball PRs
grouped_pr_body = A group of tarballs has been staged for {pr_url}.
Please review the contents of these tarballs carefully.
Merging this PR will lead to automatic ingestion of the approved tarballs to the repository {cvmfs_repo}.
Unchecked tarballs will be marked as rejected.

{tarballs}

<details>
<summary>Overview of tarball contents</summary>

{tar_overview}

</details>

{metadata}

# Template for payload overview
task_summary_payload_template =
{payload_overview}


[slack]
ingestion_notification = yes
Expand Down
Loading