-
Notifications
You must be signed in to change notification settings - Fork 50
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce GitLab CI pipeline depth #577
Conversation
Source prep is hitting that btrfs dev problem I've seen mentioned somewhere else... |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense, thanks!
I would like to add one suggestion: It would be great to not run Gitlab CI at all if |
This is a good suggestion. I wanted to get away from what we do in osbuild-composer which is gate gitlab on all the unit tests, which can take some time. But the source-prep stage, especially now that it's generating the CI config, should be in the same workflow as the GitLab trigger now. I'll move some stuff around. |
We will use this script to generate the whole CI config.
Add the init, verify, and finish jobs to the base config, so that we can generate the top-level gitlab-ci config with all the stages we need.
Sort the image configurations so that the generator is stable.
Check in the generated gitlab-ci.yml so it becomes the top-level CI config.
Generate .gitlab-ci.yml when running prepare-source. This will make sure we catch any changes to image configurations (distros, arches, image types) that aren't reflected in the CI config.
The source prep script now runs a python script that import imgtestlib, which uses newer python3 features (match). Run on ubuntu-latest so we have a newer python version and update all runners to match.
9408d05
to
bf66b67
Compare
Done and done. |
The source preparation script now generates the gitalb-ci.yml and checks that it's up to date with new image configurations. Don't run gitlab CI tests if the source prep check doesn't succeed since it might end up testing the wrong thing (distros that have been removed, for example). Also, we need build dependencies to run the script now since it compiles and runs the list-images command. Signed-off-by: Achilleas Koutsou <achilleas@koutsou.net>
bf66b67
to
ec3de25
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks.
Previously, our top-level gitlab-ci.yml included a call to
./test/scripts/configure-generators
which was used to generate a dynamic pipeline with jobs that would generate manifests for each distro/arch/image-type configuration. Then, each of those manifest generation stages in the dynamic pipeline would generate another level of dynamic pipelines based on the manifest changes.This PR reduces the depth of dynamic pipelines by 1 (to 1). The gitlab-ci.yml in the repository is replaced by the configuration that was previously dynamically generated at the first level, with the other previous top-level jobs added (
init
,verify
,finish
). A CI job makes sure that the generated config is up to date.This change has two benefits:
configure-generators
step would take 3-4 minutes and would run every time. Now it's done offline and a local run takes a few seconds. The runtime of the job in CI was mostly spent setting up the runner.Includes #576