Keep older versions in OBS

[matpen]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description

I have noticed that, whenever a new podman (or buildah) release comes out, the PPA is updated with the new one, but the old ones are removed. This makes it hard to pin versions in install scripts.

Is it possible to keep older versions in the PPA (at least a few back)?

Steps to reproduce the issue:

1. Install by pinning version: apt-get -yq install podman="1.5.0-2~ubuntu18.04~ppa5"

2. Notice that, after a release, the above code fails with
   E: Version '1.5.0-2~ubuntu18.04~ppa5' for 'podman' was not found

(Environment details omitted because not relevant)

[baude]

@lsm5 ptal

[lsm5]

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

@baude @mheon once we have a new release, we stop supporting the older ones, yes?

[mheon]

Yeah, we don't generally do backports to older versions

[lsm5]

@matpen any reason you need to pin podman version?

[matpen]

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

I think that it can be easily accomplished by using file names. At least this is what I see on the kernel ppa and gcc ppa.

For example, I used to install gcc 7 on Ubuntu 16.04 like this:

add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get -yq update
apt-get -yq install gcc-7 g++-7

@matpen any reason you need to pin podman version?

When using podman in production, things might break on system update. For example, when upgrading from 1.5.0 to 1.5.1 the default pod hostname changed: I am making use of this information inside the containers, so the "old" config will not work with the "new" podman.

For this reason I prefer to keep podman at a defined version, which I tested on my development environment, until I have the time to upgrade.

@baude @mheon once we have a new release, we stop supporting the older ones, yes?

Unfortunately I have never setup a PPA myself, so I cannot provide direct experience. But one possible way to support both approaches could be to simply deploy as follows:

• publish one podman package, which is the "default" one and always points to the latest release;
• publish one "versioned copy" of the same filename;
• on the next release, replace the default, but keep the old versioned copy.

[lsm5]

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

I think that it can be easily accomplished by using file names. At least this is what I see on the kernel ppa and gcc ppa.

For example, I used to install gcc 7 on Ubuntu 16.04 like this:

add-apt-repository ppa:ubuntu-toolchain-r/test
apt-get -yq update
apt-get -yq install gcc-7 g++-7

If upstream isn't supporting older releases, I'd rather not commit to this either.

There's another request to publish multiple PPAs where one has only the latest release, while the others have RCs/nightlies/hourlies. Perhaps you could use those (once published) to test upcoming versions?

When using podman in production, things might break on system update. For example, when upgrading from 1.5.0 to 1.5.1 the default pod hostname changed: I am making use of this information inside the containers, so the "old" config will not work with the "new" podman.

@mheon can correct me if I'm wrong, but the old behaviour was a bug. So, you should keep the new one.

[matpen]

If upstream isn't supporting older releases, I'd rather not commit to this either.

This has nothing to do with supporting old version. The requirement here is that an older version is still available through the convenience of the PPA, not that bugfixes are backported. In fact, this is exactly what should not happen (at least not without changing the version number).

There's another request to publish multiple PPAs where one has only the latest release, while the others have RCs/nightlies/hourlies. Perhaps you could use those (once published) to test upcoming versions?

This is not always an option: when the version is released, the PPAs are rotated, and it is not always possible to keep the production servers at the same pace.

the old behaviour was a bug. So, you should keep the new one.

In a perfect world, I would obviously always want the latest bugfree version. But in the real world, a new version will have incompatibilities. In this particular case the change, while definitely being an improvement, would break my containers.

Hi @matpen, just curious, do you know of any other PPAs that provide this? I don't think there's a trivial way to do this on PPA.

I just remembered that the docker project maintains exactly the envisioned setup:

[ ... ]
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
[ ... ]
sudo apt-get install docker-ce=<VERSION_STRING> docker-ce-cli=<VERSION_STRING> containerd.io

[lsm5]

Hmm, I think I can set this up, but quite likely won't be on the current launchpad ppa, but some other location where I have more control. I'll try and get back on this asap.

[afbjorklund]

I couldn't get any 1.5 nor 1.6 version to work properly on Ubuntu 16.04, so ended up having to rebuild v1.4.4 from source (since I had deleted the local .deb package)

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[rhatdan]

@lsm5 Any update on this?

[lsm5]

@lsm5 Any update on this?

Not yet, but hopefully once other distractions are outta the way ..

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@lsm5 What should we do with this?

[vrothberg]

The cross-distro packaging is now happening in the public OBS instance -> https://build.opensuse.org/project/subprojects/devel:kubic:libcontainers. The links to older package versions should be static and can hence be used in install scripts.

@lsm5 @sysrich what do you think?

[sysrich]

There are multiple approaches we could do

1. Versioned package names (eg. podman1.5 , podman1.6, etc) in the existing OBS Project.

This would be the easiest for the described usecase, but would be a significant pain for many users as there wouldn't be a simple 'apt-get podman' option..and how would people know when to move to a new version? Such things aren't a concern when talking about gcc versions but I dont think podman can be really compared to a core system compiler.

It would also be a pain for us to maintain and build, with a never ending growing amount of packages with potential build issues.

2. In RPM-land we have a concept of a 'maintenance repository' which keeps all previous versions of packages under the original name. I have NO IDEA if such things as possible with debs, but if they are OBS should be able to support the extra metadata..but it would require the modification of scripts to take benefit of such a feature, such as on openSUSE where "zypper in podman" would have to be "zypper in podman=1.5" to guarantee you only get podman version 1.5.

If I was @lsm5 I wouldn't like doing either regularly, but I could see Option 1 being useful as an exception to the norm for cases like the 'old version 1.4.4 only works on still in-support LTS distro release'

[vrothberg]

@lsm5 WDYT? Feel free to close.

[matpen]

Feel free to close.

I do not think that closing this is a good idea. In fact, from the point of view of any serious deployment, it is a very urgent issue: it pretty much makes the repos unusable. As @afbjorklund said, the only viable alternative is to build from source, which can prove tricky for some sysadmins.

Is there any technical reason why the solution offered by the docker repos is not applicable in this case?

[vrothberg]

do not think that closing this is a good idea. In fact, from the point of view of any serious deployment, it is a very urgent issue: it pretty much makes the repos unusable.

Can you elaborate a bit more on that? We provide a repository with updated and fresh builds but don't have a good approach for keeping builds around.

[lsm5]

In RPM-land we have a concept of a 'maintenance repository' which keeps all previous versions of packages under the original name. I have NO IDEA if such things as possible with debs, but if they are OBS should be able to support the extra metadata..but it would require the modification of scripts to take benefit of such a feature, such as on openSUSE where "zypper in podman" would have to be "zypper in podman=1.5" to guarantee you only get podman version 1.5.

@sysrich could you please share some info/examples on setting up a maintenance repository?

@matpen in docker's case, they probably maintain the entire repo by themselves, which I'd rather not. I guess a maintenance repo that @sysrich mentioned could address your case, we can check if that's doable for debs too. Also, for debs, I will keep the packaging sources available for older versions, even though the older debs themselves might get wiped out.

[matpen]

Can you elaborate a bit more on that? We provide a repository with updated and fresh builds but don't have a good approach for keeping builds around.

I think that the problem has been outlined sufficiently in this issue: since there is no way to pin a version, sysadmins only have the following options in production:

• upgrade podman each time they reinstall or upgrade a system (which is dangerous - see for example here)
• not use the repos and instead build from source (out of reach for some, and 10x more work in any case).

In other words, unless using the second options, sysadmins have no control on which version of podman they are running, which makes maintenance a nightmare.

in docker's case, they probably maintain the entire repo by themselves, which I'd rather not

@lsm5 could you please expand this? How are you maintaining the repo? Seems like something is preventing it to enable a feature which is otherwise available via apt-get (version pinning).

we can check if that's doable for debs too. Also, for debs, I will keep the packaging sources available for older versions, even though the older debs themselves might get wiped out.

This should be the case, as with many other repos version pinning is always available: apt-get install package=1.2.3. I would like to give more advice, but for that I need more knowledge about the particular infrastructure (see my previous question).

[rhatdan]

@lsm5 Any progress on this issue? Really should not be an upstream issue, but I guess there is no other place to report it.

[matpen]

@lsm5 Sorry I did not manage yet to find the time to work on this one. Anyway, having podman available from the official repos would make it even easier, both for the end user and for the maintainers. Great if this works out!

[siretart]

@siretart is working toward enabling podman on debian as well as ubuntu default repositories.

Done:

• https://tracker.debian.org/pkg/libpod
• https://launchpad.net/ubuntu/+source/libpod

And hopefully once those are available, I won't need to maintain debs on OBS anymore.

You may want to keep your existing work for earlier Distributions. I'm not sure whether I can commit to providing backports for debian 10 or Ubuntu 20.04 or earlier. I imagine the kernel to be particularly challenging.

I wonder whether flatpack or snapd might be an interesting option for distributing podman for earlier distro releases...

[alitvak69]

Unfortunately many people use LTS distros for ubuntu, i.e. 18.04 and 20.04 so unless ubuntu effort can be backported we would need to keep OBS or some other distro longer. IMHO snapd is not a good idea.

[vrothberg]

Reopening due to #8971

[matpen]

Following up from #8971, which has been closed in favor of reopening this issue (thank you @vrothberg).

@lsm5 I renew my intention to have a look at the OBS configuration, if you are still willing to support my progress and be patient while I try to find the time for this.

[afbjorklund]

@vrothberg : You might want to rename this to OBS then, since podman is no longer using PPA repositories ?

I think we will have to set up our own package cache, so that we don't have to rebuild from source all the time:

kubernetes/minikube#9552

[lsm5]

Following up from #8971, which has been closed in favor of reopening this issue (thank you @vrothberg).

@lsm5 I renew my intention to have a look at the OBS configuration, if you are still willing to support my progress and be patient while I try to find the time for this.

yup, let me know how I can help.

[alitvak69]

@lsm5 I see an old podman repository for centos7 in OBS. Is it indication of the things to come for other repos?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[tabic]

bump out of stale

[mheon]

This is unlikely to be fixed at OBS - it's a limitation of the platform that we can't fix ourselves. As Debian and Ubuntu start to include Podman in their own repos, we're removing them from OBS to encourage people to use the distribution's own packages, as @lsm5 doesn't have enough time to maintain packages for all these distros.

As such, I think it's unlikely that the Podman team itself will have time to fix this. However, if anyone from the community would like to step up and assist in packaging older versions to ensure we have a slow-moving stable stream, the team would be happy to offer assistance.

[buck2202]

@mheon or others, off-topic, but is there a discussion about dropping OBS build support recorded or ongoing somewhere that you could link to? I've seen that ubuntu bionic has been disabled, and saw @lsm5's blog post/brief talk in your March meeting. But as bionic LTS is still active until 2023 and has no distro package at all, I'm just looking for information about any specific headaches that prompted it (e.g. I'm aware of #9363).

For ubuntu releases, bionic with the ga 4.15.x kernel is the only active release that has functional checkpoint/restore with overlayfs, so it was disappointing to see it dropped.

Sorry to go off-topic

[afbjorklund]

A heads-up that bintray.com will be removed at the end of the month, including the old podman and cri-o rebuilds from OBS.

[lsm5]

@mheon or others, off-topic, but is there a discussion about dropping OBS build support recorded or ongoing somewhere that you could link to? I've seen that ubuntu bionic has been disabled, and saw @lsm5's blog post/brief talk in your March meeting. But as bionic LTS is still active until 2023 and has no distro package at all, I'm just looking for information about any specific headaches that prompted it (e.g. I'm aware of #9363).

For ubuntu releases, bionic with the ga 4.15.x kernel is the only active release that has functional checkpoint/restore with overlayfs, so it was disappointing to see it dropped.

Sorry to go off-topic

Well, it has been a bunch of issues mostly involving older dependencies which required me to rebuild/modify newer versions from Debian Unstable on Kubic.

Having said that, if community members would like to own packages for older distros, I can help them get started. Packaging source can be found here: https://gitlab.com/rhcontainerbot/podman/-/tree/debian . So let me know.

[lsm5]

I'm closing this. Doesn't look like anyone's working on it. Ubuntu versions 22.10 and up already have podman in their base repos and we'll stop ubuntu OBS packaging once 22.04 is released.