cosa buildextend-extensions pulls in packages from modules that are not enabled

[fidencio]

Bug Report

While it's known that rpm-ostree is not aware of modules, and that even allows us to easily build the CoreOS Extensions including packages from modules such as advanced-virt, it has some non expected side effects, such as pulling packages from modules and end up with a combination of packages to be installed which were never ever tested before or that won't work together.

The example that led to open this bug is, as shown here, the fact that libpmem 1.9.2-1.module+el8.3.1+9584+6278db68.x86_64, which comes from the pmdk module, is pulled in, while QEMU kiwi was built and tested against libpmem 1.6.1-1.el8.x86_64, which comes from AppStream.

Mind, both libpmem are part of http://cdn.redhat.com/content/beta/rhel8/8/x86_64/appstream/os/Packages/l/, which is expected, but on "dnf world" the module version would only be available if the pmdk module was enabled, while right new we see the one if the highest n-v-r being picked up.

Environment

What operating system is being used to run coreos-assembler?

[coreos-assembler]$ cat /etc/os-release 
NAME=Fedora
VERSION="33 (Container Image)"
ID=fedora
VERSION_ID=33
VERSION_CODENAME=""
PLATFORM_ID="platform:f33"
PRETTY_NAME="Fedora 33 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:33"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f33/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=33
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=33
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
VARIANT="Container Image"
VARIANT_ID=container

What operating system is being assembled?
I'm locally assembling latest RHCOS, in order to check the status of the CoreOS Extensions.

Is coreos-assembler running in Podman or Docker?
Yes, coreos-assembler is running in Podman.

If Podman, is coreos-assembler running privileged or unprivileged?
privileged, this is the command line used:

cosa() {
     env | grep COREOS_ASSEMBLER
     set -x # so we can see what command gets run
     sudo podman run --rm -ti -v ${PWD}:/srv/ --userns=host --device /dev/kvm --name cosa \
                ${COREOS_ASSEMBLER_PRIVILEGED:+--privileged}                                          \
                ${COREOS_ASSEMBLER_CONFIG_GIT:+-v $COREOS_ASSEMBLER_CONFIG_GIT:/srv/src/config/:ro}   \
                ${COREOS_ASSEMBLER_GIT:+-v $COREOS_ASSEMBLER_GIT/src/:/usr/lib/coreos-assembler/:ro}  \
                ${COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS}                                            \
                ${COREOS_ASSEMBLER_CONTAINER:-quay.io/coreos-assembler/coreos-assembler:latest} $@
     rc=$?; set +x; return $rc
 }

Expected Behavior

cosa buildextend-extensions should create the extensions content with libpmem coming from the "system" (appstrema), rather than the one from the pmdk module.

Actual Behavior

cosa buildextend-extensions will create the extensions content with libpmem with the highest n-v-r.

Reproduction Steps

1. $ cosa shell
2. $ cd src/config
3. $ cosa fetch && cosa build
4. $ cosa buildextend-extensions
5. $ tar tf ./builds/latest/x86_64/rhcos-48.84.202104160842-0-extensions.x86_64.tar | grep libpmem

Other Information

I was not able to actually find the code fetching the RPMs to compose the extensions, but I think the limitation reported here is there. A pointer to the code would be nice, at least for future reference.

If there was a way to filter out specific packages, that would be a reasonable workaround till "modularity" becomes supported by rpm-ostree.

Last but not least, this is affecting sandboxed-containers extension tests, as qemu-kiwi installation is breaking due to this issue.

[fidencio]

/cc @danilocesar @jensfr

[fidencio]

/cc @miabbott and @cgwalters, as this was discussed Yesterday with them in the #forum-coreos

[fidencio]

Now taking a look at the rpm-ostree compose extensions code, I guess it's doing the expected thing although not the correct one.

I'm not sure what's the best approach to take here, but a conflicts or excludes field would be of huge benefit for us in this case.

[jlebon]

Yes, the underlying code is in rpm-ostree: https://github.com/coreos/rpm-ostree/blob/b2b3795df126927236bbff81a560c95c25a508b1/src/app/rpmostree-compose-builtin-tree.cxx#L1432

cosa buildextend-extensions should create the extensions content with libpmem coming from the "system" (appstrema), rather than the one from the pmdk module.

Ahhh interesting. I somehow hadn't realized til now that the AppStream mixed modular with non-modular packages. In Fedora we've been mostly able to work around this just by disabling the modular repos.

But then... ahhh right, in RHCOS today we're already getting a bunch of container-related packages from modules that way.

So anyway, long-term we obviously need coreos/rpm-ostree#1435. Short-term, we should be able to hack around this adding an exclude like libpmem*module* to the rhel-8-appstream repo:

$ git -C src/config diff
diff --git a/rhel8.repo b/rhel8.repo
index 9991bec..4ce3967 100644
--- a/rhel8.repo
+++ b/rhel8.repo
@@ -13,7 +13,7 @@ baseurl=http://.../content/beta/rhel8/8/$basearch/appstrea
 enabled=1
 gpgcheck=0
 # Exclude toolbox to make sure we use coreos/toolbox from rhaos repo
-exclude=conmon toolbox
+exclude=conmon toolbox libpmem*module*

 [rhel-8-nfv]
 baseurl=http://.../content/beta/rhel8/8/x86_64/nfv/os/
$ cosa buildextend-extensions
...
$ jq .extensions.manifest.libpmem builds/latest/x86_64/meta.json
"1.6.1-1.el8.x86_64"

But actually, now that we have the libdnf bindings for it, coreos/rpm-ostree#1435 might not even be that hard and will unblock a bunch of other things. I'll try to take a crack at it soon-ish unless someone else beats me to it. :)

[fidencio]

That simply does work as expected, thanks @jlebon.

On a side note, I tried that in the morning (or something quite similar) and I was still getting the wrong libpmem, and I sincerely think that was a typo from my side. :-/

[jlebon]

OK cool! Let's close this since the real issue here is already tracked at coreos/rpm-ostree#1435.