Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

package layering fails on f32 base with modular packages #525

Closed
grantral opened this issue Jun 6, 2020 · 11 comments · Fixed by coreos/fedora-coreos-config#464
Closed
Labels
component/rpm-ostree fedora-test-day Issues filed during Fedora Test Day

Comments

@grantral
Copy link

grantral commented Jun 6, 2020

Steps to reproduce

  1. Install last stable stream; fedora-coreos-31.20200517.3.0-live.x86_64.iso
  2. $ sudo rpm-ostree install buildah -r
  3. $ sudo rpm-ostree install flatpak -r
  4. $ sudo rpm-ostree rebase fedora/x86_64/coreos/next -r

Expected behaviour

The system successfully reboots and runs on the next stream.

Actual behaviour

Receiving objects: 99% (9825/9836) 1.0 MB/s 523.6 MB 
Receiving objects: 99% (9825/9836) 1.0 MB/s 523.6 MB... done
Checking out tree 08040be... done
Enabled rpm-md repositories: updates fedora
Updating metadata for 'updates'... done
rpm-md repo 'updates'; generated: 2020-06-05T01:26:33Z
Updating metadata for 'fedora'... done
rpm-md repo 'fedora'; generated: 2020-04-22T22:22:36Z
Importing rpm-md... done
Forbidden base package replacements:
  policycoreutils 3.0-2.module_f32+7989+651e8914 -> 3.0-2.fc32 (fedora)
This likely means that some of your layered packages have requirements on newer or older versions of some base packages. Doing `rpm-ostree cleanup -m` and `rpm-ostree upResolving dependencies... done
error: Some base packages would be replaced

System configuration

Ignition config

variant: fcos
version: 1.1.0
passwd:
  users:
    - name: core
      password_hash: $y$j9T$oUoFYoBD1fmR12XKbZOjU.$7gbf0M2XCYsUY.Njp9deA3leG/p1scY2a30cvBLs6J7
      ssh_authorized_keys:
        - ssh-rsa AAAAB3NzaC1yc...
storage:
  files:
    - path: /etc/ssh/sshd_config.d/03-enable-passwords.conf
      mode: 0644
      contents:
        inline: |
          # Fedora CoreOS disables SSH password login by default.
          # Enable it.
          # This file must sort before 04-disable-passwords.conf.
          PasswordAuthentication yes

rpm-ostree status

State: idle
Deployments:
● ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200517.3.0 (2020-06-01T17:15:29Z)
                BaseCommit: 967b7b8d624e6d10ff51c2e81ef198fae966c567ac2e9b479771c693d0987949
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
           LayeredPackages: buildah flatpak

  ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 31.20200517.3.0 (2020-06-01T17:15:29Z)
                BaseCommit: 967b7b8d624e6d10ff51c2e81ef198fae966c567ac2e9b479771c693d0987949
              GPGSignature: Valid signature by 7D22D5867F2A4236474BF7B850CB390B3C3359C4
           LayeredPackages: buildah

Notes

  1. rpm-ostree suggests using rpm-ostree up, but this command does not exist;
  2. Output rpm-ostree upResolving dependencies... done is merged.
@dustymabe
Copy link
Member

It turns out this has nothing to do with the rebase. If you start from a fresh install of next you also have this problem with package layering. I think part of the problem is that we are bumping our sources for FCOS with fedora modular repos enabled but a booted FCOS system has them disabled.

If you edit /etc/yum.repos.d/fedora-modular.repo and /etc/yum.repos.d/fedora-updates-modular.repo to have enabled=1 for the first entry in each file the overlay will work fine.

Maybe we should stop enabling the modular repos when we build FCOS?

Here's the diff I get when I disable those repos and do a testing-devel build:

--- ./manifest-lock.x86_64.json 2020-06-06 23:48:24.696660245 -0400
+++ /var/b/shared/assembler/fedora/lockme       2020-06-07 00:07:59.305425327 -0400
@@ -70,7 +70,7 @@
       "evra": "1.0.8-2.fc32.x86_64"
     },
     "c-ares": {
-      "evra": "1.16.0-1.module_f32+8723+aeaf79ed.x86_64"
+      "evra": "1.15.0-5.fc32.x86_64"
     },
     "ca-certificates": {
       "evra": "2020.2.40-3.fc32.noarch"
@@ -234,9 +234,6 @@
     "expat": {
       "evra": "2.2.8-2.fc32.x86_64"
     },
-    "fedora-coreos-pinger": {
-      "evra": "0.0.4-1.module_f32+6507+a4e4adf6.x86_64"
-    },
     "fedora-gpg-keys": {
       "evra": "32-2.noarch"
     },
@@ -889,7 +886,7 @@
       "evra": "2:1.9.3-1.fc32.x86_64"
     },
     "policycoreutils": {
-      "evra": "3.0-2.module_f32+7989+651e8914.x86_64"
+      "evra": "3.0-2.fc32.x86_64"
     },
     "polkit": {
       "evra": "0.116-7.fc32.x86_64"
@@ -1088,7 +1085,7 @@
     }
   },
   "metadata": {
-    "generated": "2020-06-06T21:07:37Z",
+    "generated": "2020-06-07T04:07:55Z",
     "rpmmd_repos": {
       "fedora": {
         "generated": "2020-04-22T22:22:36Z"
@@ -1096,14 +1093,8 @@
       "fedora-coreos-pool": {
         "generated": "2020-06-05T21:39:01Z"
       },
-      "fedora-modular": {
-        "generated": "2020-04-22T21:03:13Z"
-      },
       "fedora-updates": {
         "generated": "2020-06-05T01:26:33Z"
-      },
-      "fedora-updates-modular": {
-        "generated": "2020-06-05T04:14:06Z"
       }
     }
   }

I'm guessing fedora-coreos-pinger isn't set up to have a default stream yet (should be easy to fix).

@dustymabe dustymabe changed the title Stable stream can not switch to the next stream package layering fails on f32 base with modular packages Jun 7, 2020
@dustymabe
Copy link
Member

btw thanks @grantral for taking part in the test day!

@lucab lucab added component/rpm-ostree fedora-test-day Issues filed during Fedora Test Day labels Jun 8, 2020
@jlebon
Copy link
Member

jlebon commented Jun 8, 2020

Maybe we should stop enabling the modular repos when we build FCOS?

Yeah... a lot of backstory here. It essentially comes down to rpm-ostree not being modularity aware (coreos/rpm-ostree#1435, which is blocked on rpm-software-management/libdnf#874).

If we can stop using modular repos now that the Rust packages are in the regular repos again, sure. Otherwise I think we could tweak rpm-ostree to work in this case by adding more hacks on top of the existing hacks.

@dustymabe
Copy link
Member

@zonggen @rfairley - could we get fedora-coreos-pinger into the default repos so that we can drop the modularity repos from our composes?

@zonggen
Copy link
Member

zonggen commented Jun 8, 2020

@dustymabe I'm not entirely sure the procedure to get pinger into default repos but I can create a card for it and work on that

@dustymabe
Copy link
Member

@zonggen i think now that rust builds in Fedora no longer use modules it should be as simple as bumping the release in the rpm spec and doing a new round of builds. Can you work with @rfairley to make that happen? You can re-use this issue if you like.

@rfairley
Copy link

rfairley commented Jun 8, 2020

Yes, to fix this immediately we can rebuild https://src.fedoraproject.org/rpms/fedora-coreos-pinger at the current upstream version. Updating the release number https://src.fedoraproject.org/rpms/rust-fedora-coreos-pinger/blob/master/f/rust-fedora-coreos-pinger.spec#_9, rebuilding with koji build and then tagging the build into F32 should be what's needed. For the tag, ask Igor (ignatenkobrain) in #fedora-rust to do this (I think this is still the case it needs to go through Igor - soon we'll be able to do it ourselves).

@zonggen, let's coordinate tomorrow - can help if any issues come up. Sorry to have seen this notification late today.

@rfairley
Copy link

rfairley commented Jun 9, 2020

Turns out we do have a build not using modular repos for rust-fedora-coreos-pinger: https://koji.fedoraproject.org/koji/buildinfo?buildID=1452289, but no bodhi update in f32. Currently unable to log in to Bodhi to create the update, I think due to the datacenter move (https://hackmd.io/vDwoayVLQ8yjyDk3PDvC8A?view#Week-14-2020-06-08--gt-2020-06-14). We should be able to get around this by adding an override for fedora-coreos-pinger-0.0.4-3.fc32 without the Bodhi update initially, but will try tomorrow if Bodhi will be back up, so the Bodhi update can be created first.

rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 9, 2020
@rfairley
Copy link

rfairley commented Jun 9, 2020

Turns out we do have a build not using modular repos for rust-fedora-coreos-pinger: https://koji.fedoraproject.org/koji/buildinfo?buildID=1452289

Creating a bodhi update for this, Bodhi did not find the build, I think due to it having an f33 tag but not f32 (saw on the page before Koji went offline). Trying an override locally with coreos-assembler and testing-devel gives error: Couldn't find locked package 'fedora-coreos-pinger-0.0.4-3.fc32.x86_64' . For now I've opened https://src.fedoraproject.org/rpms/rust-fedora-coreos-pinger/pull-request/2. Koji is down today for another 8 or so hours - will try rebuilding with this later and side-tagging into F32 with the new build.

rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 9, 2020
Fasttrack the non-modular fedora-coreos-pinger build, removing our
dependency on modular content. Part of fixing
coreos/fedora-coreos-tracker#525.
rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 9, 2020
Remove modular repos, as we no longer require sourcing modular
content.

Fixes: coreos/fedora-coreos-tracker#525
rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 9, 2020
Fast-track the non-modular fedora-coreos-pinger build, removing our
dependency on modular content. Part of fixing
coreos/fedora-coreos-tracker#525.

No Bodhi update is added yet due to the migration currently
active in Fedora, but should be added as soon as possible.
rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 9, 2020
Remove modular repos, as we no longer require sourcing modular
content.

Fixes: coreos/fedora-coreos-tracker#525
rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 9, 2020
Remove modular repos, as we no longer require sourcing modular
content.

Fixes: coreos/fedora-coreos-tracker#525
jlebon added a commit to jlebon/rpm-ostree that referenced this issue Jun 9, 2020
While investigating
coreos/fedora-coreos-tracker#525, I realized
that libsolv does have a flag which allows us to express that our base
packages shouldn't be upgraded: `SOLVER_LOCK`. This then lets us not
have to sanity-check the solution post-solve for "forbidden
replacements", and lets libsolv/libdnf print exactly what the conflict
is.
@jlebon
Copy link
Member

jlebon commented Jun 9, 2020

Otherwise I think we could tweak rpm-ostree to work in this case by adding more hacks on top of the existing hacks.

I retract this statement. When trying rpm-ostree install flatpak on top of coreos/rpm-ostree#2125, it becomes clearer what the problem is:

[root@cosa-devsh ~]# rpm-ostree install flatpak
Checking out tree 08040be... done
Enabled rpm-md repositories: fedora-cisco-openh264 updates fedora
rpm-md repo 'fedora-cisco-openh264' (cached); generated: 2020-03-17T20:10:45Z
rpm-md repo 'updates' (cached); generated: 2020-06-07T18:20:54Z
rpm-md repo 'fedora' (cached); generated: 2020-04-22T22:22:36Z
Importing rpm-md... done
Resolving dependencies... done
error: Could not depsolve transaction; 1 problem detected:
 Problem: conflicting requests
  - package flatpak-1.6.3-1.fc32.x86_64 requires (flatpak-selinux = 1.6.3-1.fc32 if selinux-policy-targeted), but none of the providers can be installed
  - package flatpak-1.6.3-1.fc32.i686 requires libsoup-2.4.so.1, but none of the providers can be installed
  - package flatpak-selinux-1.6.3-1.fc32.noarch requires policycoreutils-python-utils, but none of the providers can be installed
  - package policycoreutils-python-utils-3.0-2.fc32.noarch requires python3-policycoreutils = 3.0-2.fc32, but none of the providers can be installed
  - package python3-policycoreutils-3.0-2.fc32.noarch requires policycoreutils = 3.0-2.fc32, but none of the providers can be installed
  - cannot install both policycoreutils-3.0-2.fc32.x86_64 and policycoreutils-3.0-2.module_f32+7989+651e8914.x86_64

So we have flatpak -> flatpak-selinux -> policycoreutils-python-utils -> python3-policycoreutils -> python3-policycoreutils -> policycoreutils. We only have the last one on the system, and of course not the Python ones. And since the one in the base layer is the modular one, and we have modular repos disabled, it can't find the matching ones. So this is a variant of coreos/rpm-ostree#415.

rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 10, 2020
Remove modular repos from the list in this file, as we no longer
require sourcing modular content. Part of:
coreos/fedora-coreos-tracker#525
rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 10, 2020
Add the recent non-modular fedora-coreos-pinger build to
overides, removing our dependency on modular content. Part
of fixing coreos/fedora-coreos-tracker#525.

Overriding is necessary as there is no Bodhi update yet.
The Bodhi update is pending a Fedora admin building with the
side tag to make the package available in F32. Permissions for
us to create the side tag are also pending, see:
https://pagure.io/releng/issue/9229. Once either of the two
pending above are completed, we'll be able to create and add
the Bodhi update here (see TODO in the manifest).
rfairley pushed a commit to rfairley/fedora-coreos-config that referenced this issue Jun 10, 2020
Remove modular repos from the list in this file, as we no longer
require sourcing modular content. Part of:
coreos/fedora-coreos-tracker#525
dustymabe pushed a commit to coreos/fedora-coreos-config that referenced this issue Jun 10, 2020
Add the recent non-modular fedora-coreos-pinger build to
overides, removing our dependency on modular content. Part
of fixing coreos/fedora-coreos-tracker#525.

Overriding is necessary as there is no Bodhi update yet.
The Bodhi update is pending a Fedora admin building with the
side tag to make the package available in F32. Permissions for
us to create the side tag are also pending, see:
https://pagure.io/releng/issue/9229. Once either of the two
pending above are completed, we'll be able to create and add
the Bodhi update here (see TODO in the manifest).
dustymabe pushed a commit to coreos/fedora-coreos-config that referenced this issue Jun 10, 2020
Remove modular repos from the list in this file, as we no longer
require sourcing modular content. Part of:
coreos/fedora-coreos-tracker#525
dustymabe pushed a commit to dustymabe/fedora-coreos-config that referenced this issue Jun 11, 2020
Add the recent non-modular fedora-coreos-pinger build to
overides, removing our dependency on modular content. Part
of fixing coreos/fedora-coreos-tracker#525.

Overriding is necessary as there is no Bodhi update yet.
The Bodhi update is pending a Fedora admin building with the
side tag to make the package available in F32. Permissions for
us to create the side tag are also pending, see:
https://pagure.io/releng/issue/9229. Once either of the two
pending above are completed, we'll be able to create and add
the Bodhi update here (see TODO in the manifest).
dustymabe pushed a commit to dustymabe/fedora-coreos-config that referenced this issue Jun 11, 2020
Add the recent non-modular fedora-coreos-pinger build to
overides, removing our dependency on modular content. Part
of fixing coreos/fedora-coreos-tracker#525.

Overriding is necessary as there is no Bodhi update yet.
The Bodhi update is pending a Fedora admin building with the
side tag to make the package available in F32. Permissions for
us to create the side tag are also pending, see:
https://pagure.io/releng/issue/9229. Once either of the two
pending above are completed, we'll be able to create and add
the Bodhi update here (see TODO in the manifest).

(cherry picked from commit af7003d)
@dustymabe
Copy link
Member

The fix for this went into testing stream release 32.20200601.2.2.

The fix for this went into stable stream release 32.20200601.3.0.

@dustymabe dustymabe removed the status/pending-testing-release Fixed upstream. Waiting on a testing release. label Jun 19, 2020
dustymabe pushed a commit to dustymabe/fedora-coreos-config that referenced this issue Jul 10, 2020
Remove modular repos from the list in this file, as we no longer
require sourcing modular content. Part of:
coreos/fedora-coreos-tracker#525

(cherry picked from commit 74c98c2)
dustymabe pushed a commit to dustymabe/fedora-coreos-config that referenced this issue Jul 10, 2020
Remove modular repos from the list in this file, as we no longer
require sourcing modular content. Part of:
coreos/fedora-coreos-tracker#525

(cherry picked from commit 74c98c2)
dustymabe pushed a commit to coreos/fedora-coreos-config that referenced this issue Jul 10, 2020
Remove modular repos from the list in this file, as we no longer
require sourcing modular content. Part of:
coreos/fedora-coreos-tracker#525

(cherry picked from commit 74c98c2)
jlebon added a commit to jlebon/rpm-ostree that referenced this issue Aug 26, 2020
For the Fedora CoreOS extensions work, when layering packages, we need
to be able to tell libsolv to pick the packages which will go with the
base packages. IOW, it needs to know that the base packages shouldn't be
uninstalled.

While investigating
coreos/fedora-coreos-tracker#525, I realized
that libsolv does have a flag which allows us to express this:
`SOLVER_LOCK`.

This then allows libsolv to choose the right package for us (if found).
And in the case where it can't find a matching package, libsolv itself
will print exactly what the conflict is, allowing us to avoid
sanity-checking the solution post-solve for "forbidden replacements".

Update submodule: libdnf
jlebon added a commit to jlebon/rpm-ostree that referenced this issue Aug 27, 2020
For the Fedora CoreOS extensions work, when layering packages, we need
to be able to tell libsolv to pick the packages which will go with the
base packages. IOW, it needs to know that the base packages shouldn't be
uninstalled.

While investigating
coreos/fedora-coreos-tracker#525, I realized
that libsolv does have a flag which allows us to express this:
`SOLVER_LOCK`.

This then allows libsolv to choose the right package for us (if found).
And in the case where it can't find a matching package, libsolv itself
will print exactly what the conflict is, which is more informative than
the "forbidden replacements" error we currently print out.

Update submodule: libdnf
openshift-merge-robot pushed a commit to coreos/rpm-ostree that referenced this issue Aug 28, 2020
For the Fedora CoreOS extensions work, when layering packages, we need
to be able to tell libsolv to pick the packages which will go with the
base packages. IOW, it needs to know that the base packages shouldn't be
uninstalled.

While investigating
coreos/fedora-coreos-tracker#525, I realized
that libsolv does have a flag which allows us to express this:
`SOLVER_LOCK`.

This then allows libsolv to choose the right package for us (if found).
And in the case where it can't find a matching package, libsolv itself
will print exactly what the conflict is, which is more informative than
the "forbidden replacements" error we currently print out.

Update submodule: libdnf
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/rpm-ostree fedora-test-day Issues filed during Fedora Test Day
Projects
None yet
6 participants