Consider creating floating OS tags for use by repos which aren't dotnet/runtime

[mconnew]

In PR 1274 the following statement is made:

We use version specific references in infra to ensure that our CI builds are reliable. One can imagine using floating OS tags (such as debian-oldest and debian-latest), however such an approach would be guaranteed to break our build. We know that since we often see build and test breaks that need addressing in PRs where we update build and test images.

While this statement may be true for the dotnet/runtime repo, it's not for others. The WCF repo hasn't ever had a build break due to incompatibilities from a Linux distro update. With only one exception (and that's Windows only), we have zero native platform calls in our code base. If there ever is an implementation incompatibility with a new OS distro, it will get discovered and fixed in the dotnet implementation in the runtime repo before we ever use that distro.

What we have had happen multiple times is our build break due to a Linux distros being retired in the build system and we weren't informed that was going to happen.

Having someone who is working on retiring older container images for unsupported distros having to reach out to each repo that is still using it, and then having someone on that team make a change, submit a PR, verify everything is still working, then merge it, that's all a manual process with lots of room for mistakes to be made. It also adds extra non-productive work which is a lot more disruptive and costly for smaller dev teams.

Instead of a concept of latest, (eg debian-latest), what about having a tag for the current distro? Eg for debian it would be something like debian-current. The current tag would get moved to a new distro once it's been verified that the dotnet/runtime repo is passing all its tests (or at least any failures aren't due to the new distro) on a new distro version. Maybe even wait a month after that to allow any repos with an infrequent darc update period will pick up a newer build of dotnet if any product fixes were needed. Any repo that's sensitive to distro changes and want to manually and intentionally move forward can stick to the current distro version named tags, but repos that are happy with the latest that dotnet/runtime has resolved any issues for can use the -current suffix? It will cut down on build system maintenance for teams like WCF where developer time is already stretched very thin.

[mthalman]

cc @richlander

[richlander]

This is good topic. Yes, the content was written with a dotnet/runtime bias.

We could consider this. I certainly believe you that Linux breaks are low. The major point of the bottom part of the product is to insulate most libraries and apps from pain. If we did this, there would be a risk of CI breakage, but it likely isn't that high. It would be very similar to people who use ubuntu-latest for GitHub Actions, which I assume is a lot.

The key question is how many repos would use this model. Assuming we agree to go this direction, is it worth the effort to maintain the model as you propose. I think if we got a half-dozen repos to sign up for that, it would make sense.

[richlander]

After a tiny bit more thought, I think we'd want to take a page from our build images: https://github.com/dotnet/runtime/blob/main/docs/workflow/using-docker.md#the-official-runtime-docker-images

I propose a model like this:

• Offer convenience tags for Alpine and Ubuntu (to start; limit scope).
• Add a .NET version to the tags to avoid undesirable coupling between release branches.
• Offer just one tag per distro, likely aligning with latest.

I'm thinking of something like: alpine-net10.0.

Alternatively, is there a reason why Arcade-provided templates are not updating these tags for you? That's another way to solve this problem.

[mconnew]

If there's a way to get something like darc to update them, that works for me too. I don't know enough about the inner workings of darc, whether you can have a concept of a version of something which isn't a package, but if it could automatically update our versions file with a string which could be used as the basis of a container name, then that would be a workable solution.

We do have a property to specify which .NET version to publish our tests to for sending to Helix, so we could use that value as part of the container tag. Eg we would do something like this:

    <HelixTargetQueue Include="(Debian-$(XUnitPublishTargetFramework))Ubuntu.2204.Amd64@mcr.microsoft.com/dotnet-buildtools/prereqs:debian-$(XUnitPublishTargetFramework)" />

I'm happy with either option, or some other solution, my main goal is to not have to update things because a distro version is out of support. We really only run on a broad range of distros because in the early days of .NET Core, the WCF tests caught subtle differences in networking/security related behaviors between distros that corefx didn't as our tests covered a wider variety of scenarios (back when different libcurl installations could be compiled with different options). We haven't caught a problem which the runtime missed in a long time now, so it's more of a sanity check in case a customer has a problem with a specific distro we can be confident it's something specific to their setup rather than a fundamental problem.

[richlander]

@mthalman @mmitche -- Who are the best folks to consider what a good model would look like? This isn't pressing.

I agree with @mconnew that we should consider aligning ease-of-use and reliability concerns. Not every repo is as "concerning" as dotnet/runtime. On the flip-side, old container images are not serving any purpose for us.

[MichaelSimons]

A PR flow model would be broadly useful. Most product repos would be able to utilize this. This would reduce manual work while keeping more repos on the latest.

[richlander]

I am guessing that repos split via MSBuild vs yaml, w/how the container images are specified. I am also guessing that the repos to start with would be the MSBuild ones. Is that right?

[mmitche]

@MichaelSimons When you talk about a PR model, do you mean something that would automatically move from debian-11 to debian-12? or from specific image builds of debian-11?

I generally favor a tag model. OS version specific tags (debian-11), OS family specific tags (debian-latest), and specific image references where needed (e.g. a break occurs) over a PR model. Our existing PR model that uses darc is frequently an invitation to stay on old versions of dependencies, rather than upgrade them. This is typically because of a couple factors:

• Subscriptions have to be managed properly for many target+repo branch+dependency combinations. This is a challenge already.
• Stale and ignored PRs are common. Repos that do not get a lot of attention are frequently just left with open PRs. Auto-merge is not an option here. We shouldn't be making changes to repositories without human signoff.

Consider the 'simplest' dependency we have sitting around that flows widely, arcade. There are repositories with ancient versions of arcade sitting around simply because the branch or repo doesn't get a lot of attention. Other non-arcade dependencies tend to be much worse.

I think we'd be inviting less uniformity and up-to-date-ness (important for security!) than if we encouraged folks to just take a latest tag.

The other thing is that a tag model is already implementable in the current system. Maestro changes to update images are non-trivial and is a huge feature ask.

[MichaelSimons]

@MichaelSimons When you talk about a PR model, do you mean something that would automatically move from debian-11 to debian-12? or from specific image builds of debian-11?

I am referring to opening a PR to upgrade between OS versions. This is when there are risks associated with the upgrade. We already have OS version specific tags. The tags get updated frequently today via automation to pick up the latest security fixes. Repos automatically get the updates because of the "floating" nature of the tags. This work well.

[mthalman]

I am referring to opening a PR to upgrade between OS versions. This is when there are risks associated with the upgrade. We already have OS version specific tags. The tags get updated frequently today via automation to pick up the latest security fixes. Repos automatically get the updates because of the "floating" nature of the tags. This work well.

I'm sure this could be done using Renovate. That wouldn't require any infrastructure work on our part beyond a configuration file in the repo. As long as we defined the tags in such a way that Renovate can interpret the version-ness, it might just work.

[richlander]

We should probably have a meeting about this. I suspect that there is a lot of context that isn't shared across everyone. Isn't there a "CI Council" group. Can we crash your meeting?

We'd like to float this idea with you. I can tell that there are some fixed views on this, but I imagine we can design a better system if we attag it together.