Testing `dnf4 --transient`

[evan-goode]

The DNF 4 bootc plugin needs integration tests, and ci-dnf-stack is not equipped to test things meant for OSTree systems. Some unanswered questions are:

• Where should test source code live? IMO, as much as possible should be in this upstream repository.
• What testing framework should we use?
• Where should the tests run (GHA, CoreOS Jenkins, Testing Farm)?
• Should we write the tests from scratch, or base them on existing rpm-ostree and/or bootc tests?
• Will any of the answers to the above need to change as development of the bootc plugin progresses?

[evan-goode]

cc @cgwalters

[cgwalters]

Where should the tests run (GHA, CoreOS Jenkins, Testing Farm)?

I think Testing Farm is likely the most practical. (Which also points to the "testing framework" issue, we've tried to make tmt work)

Should we write the tests from scratch, or base them on existing rpm-ostree and/or bootc tests?

There may be some bits that could be copied, but I'm not sure how much. Probably the most interesting thing here is that at least on the bootc side we build a container with the updated bootc and test that.

---

In the more medium term I think we may try to reprioritize "run bootc/ostree in a container" which would dramatically simplify this story.

[evan-goode]

For now, let's follow what bootc has done upstream [1] and use TMT/Testing Farm with bootc install-to-existing-root.

[1] bootc-dev/bootc#1249.

[ppisar]

RHEL QE people already have some tests. Problem with Testing Farm is that it still does not support installing custom packages (i.e. a dnf build to test) into a bootc system.

[evan-goode]

Problem with Testing Farm is that it still does not support installing custom packages (i.e. a dnf build to test) into a bootc system.

Is this the reason why we can't use the bootc TMT provisioning plugin? That plugin expects a container-image parameter, so why couldn't we build our ci-dnf-stack container with ./container-test as usual (but based on a bootc base image) and use that image?

[cgwalters]

This is a hugely complex topic. One thing I personally find really painful about Testing Farm is that it doesn't by default at a low level solve this problem of injecting code-under-test; it's the responsibility of higher level driving tools to do so. For example https://packit.dev does this.

The way I'd personally much prefer things to work relates to https://issues.redhat.com/browse/TFT-2751 - in that flow, one would build a derived bootc container image with the content-under-test (e.g. a dockerfile that's just FROM <base> RUN dnf -y upgrade dnf basically) and then provision it using the to-existing-root flow above.

However, that from-a-registry path does complicate local testing; there's some inherent friction and duplication in requiring pushing containers-under-test to a local registry (space duplication and TLS/networking considerations).

Is this the reason why we can't use the bootc TMT provisioning plugin?

I think you (or anyone) could, but it does require nested virt today. That path makes sense in some cases, but ultimately the path I want to streamline more looks closer to virt-manager/virt-manager#739