Build ostree images that depend on commits #37
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
achilleas-k
force-pushed
the
test/build-ostree
branch
11 times, most recently
from
ba6a833
to
2605c7d
Compare
achilleas-k
force-pushed
the
test/build-ostree
branch
7 times, most recently
from
256fe03
to
cb1d088
Compare
achilleas-k
force-pushed
the
test/build-ostree
branch
8 times, most recently
from
1079e69
to
c3d945d
Compare
achilleas-k
changed the title
achilleas-k
force-pushed
the
test/build-ostree
branch
from
c3d945d
to
a9ceeb9
Compare
thozza
reviewed
Jul 31, 2023
achilleas-k
force-pushed
the
test/build-ostree
branch
from
6fa68db
to
f03bb32
Compare
|
Found a bug that only generates one ostree build job for each distro. Fixing now. |
Moved common test functions to a new (single-file) importable python module called imgtestlib. These will be shared between the two config generators, the current one and the one for ostree test generation.
New dynamic pipeline generator similar to test/generate-build-config but for image types that require pulling an ostree commit. The script filters out configs that do not specify a dependency. The script first collects all dependency build configurations (for example edge-container, empty.json) and generates their manifests to calculate the manifest ID. The manifest ID is used to find the corresponding container in the gitlab registry, which should have already been built. For each config that specifies a dependency (for example test/configs/edge-ostree-pull-empty.json) the container for the dependency is pulled from the gitlab registry and launched in order to generate manifests with ostree commit resolution enabled. Each config with a dependency is rewritten with the URL for the corresponding container. Once the manifest generation is done, the builds are filtered in the same was as they are in test/generate-build-config: for any manifest ID not found in the build cache, a job is added to the pipeline to build it. The script does not yet write the gitlab pipeline file for the builds. Note that this pipeline generator can only configure builds for the architecture that it is running on since it needs to run containers for a given architecture when generating manifests.
Use a context manager to start containers and stop them when exiting the context. Maintain an internal map of container names to ports to make sure we don't run the same container multiple times.
Instead of a hard-coded skip list, decide what to skip based on the config itself: if an ostree URL is specified, skip the test, unless `skip_ostree_pull=False`.
When generating manifests to set up build configs, use temporary directories to avoid contamination from previous builds, especially when running locally (outside CI).
Write the build configurations for the dynamic pipeline. The $PARENT_PIPELINE_ID will need to be set in the main .gitlab-ci.yml so that the child pipeline can access artifacts from the parent pipeline. The child job configs aren't complete yet. It still needs the command to start the container for the build job and the config needs to be written as an artifact for the child job to read.
Read and return the full manifest data and their IDs when generating manifests. This moves the ID calculation outside the filter function.
The generate-ostree-build-config script writes build configs in a directory defined by the second command line argument. Each config is associated with a child build job and defines the ostree remote URL and ref to pull the ostree commit from, matching the container pull and run command in the pipeline configuration.
Install dependencies in the test job definition and not the build script itself. This makes the build script distro agnostic, in case we switch certain runners to run in the shell runner for simplicity. The build configs for the first child pipeline were also changed so that the script is an array of commands instead of a multiline script.
Added new stages for the ostree build test generator and the dynamic pipeline trigger jobs. The generator script can only generate job configs for the architecture it is running on so we need to run a separate job for each architecture.
Set PYTHONUNBUFFERED so that script output is printed to the CI log without buffering/batching. Drop unneeded print flushing.
Make the linter happy.
While the first child pipeline generator can generate configs for all architectures, split it just like we do with the ostree pipeline generator to share the work and speed up the manifest generation a bit.
Document how dynamic pipelines are created for our image build testing in CI.
Link to example project that demonstrates the dynamic pipeline generation feature of GitLab CI. This can help understand how we generate the pipelines in this repo and can also be used to test configuration changes quickly.
achilleas-k
force-pushed
the
test/build-ostree
branch
from
f03bb32
to
bc31f93
Compare
Fixed and amended. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds image building to CI for image types that require pulling an ostree commit. The whole process of how this works is described in the update test README.
Copying the description of the relevant stage here:
The new stage of the workflow runs the
./test/generate-ostree-build-configscript. It has the same purpose as the config generator for the regular builds, but it sets up ostree containers to serve commits to generate manifests for the image types that depend on them.The config generator:
./cmd/gen-manifeststool.dependspart of a build config .iot-ostree-pull-emptywill cause a manifest to be generated foriot-containerwith theemptyconfig for all distros.iot-ostree-pull-emptyis mapped in theconfig-mapto the image typesiot-ami,iot-installer,iot-raw-image, andiot-vsphere. This will create four configs for each distro, one for each image type, that will all have ostree options to pull an ostree commit from aniot-containerof the same distro.-skip-noconfigflag, which means that any image type not defined in the map is skipped../test/cases/build-image.shscript for a given distro, image type, and config file.NullConfig, which is a simple shell runner that exits successfully. This is required because the child pipeline config cannot be empty.