Skip to content

CoreCLR packages are not servicing ready #112710

Closed
#113020
Closed
CoreCLR packages are not servicing ready#112710
#113020
Assignees
amanasifkhalid
Labels
area-Infrastructure-coreclrOnly use for closed issuesin-prThere is an active PR which will close this issue when it is merged
Milestone
10.0.0

Description

@ericstj
ericstj
opened on Feb 19, 2025

Recently identified in #111444

All packages under https://github.com/dotnet/runtime/tree/7bb850f5f660e815aefe5a0937c6c29635820af8/src/coreclr/.nuget are not set up to ship in servicing.

If they're meant to be enabled/disabled incrementally then a process around versioning and enabling needs to be documented and followed for servicing changes.

If they're meant to always ship, then changes to the build need to be made to enable them and consumers need to be set up to react to those regular changes.

Activity

  1. ghost added
    area-Infrastructure-coreclrOnly use for closed issues
    on Feb 19, 2025
  2. github-project-automation
    added this to Runtime Infraon Feb 19, 2025
  3. dotnet-policy-service
    added
    untriagedNew issue has not been triaged by the area owner
    on Feb 19, 2025

  4. dotnet-policy-service commented on Feb 19, 2025

    @dotnet-policy-service
    dotnet-policy-servicebot
    on Feb 19, 2025 – with Dotnet Policy Service
    Contributor

    Tagging subscribers to this area: @hoyosjs
    See info in area-owners.md if you want to be subscribed.

  5. ericstj commented on Feb 20, 2025

    @ericstj
    ericstj
    on Feb 20, 2025
    MemberAuthor

    This may apply for packages other than those in coreclr/.nuget. Those are just the ones we noticed. It makes sense to do an audit of all packages maintained by runtime to see if they have a process for servicing. That way you can discover gaps before the day they need to be serviced.

  6. hoyosjs commented on Feb 21, 2025

    @hoyosjs
    hoyosjs
    on Feb 21, 2025
    Member

    @ericstj - some of them are meant to be published every time. What stopped producing them/marking them as shipping?

  7. ericstj commented on Feb 21, 2025

    @ericstj
    ericstj
    on Feb 21, 2025
    MemberAuthor

    They've always been that way if you look at the publishing history, for example https://www.nuget.org/packages/runtime.win-x64.Microsoft.NETCore.ILDAsm

    I suspect if you want them to build with every release you might need to determine if that would have any side-effects - make sure the packages are setup to flow / version correctly / etc.

  8. JulieLeeMSFT
    assigned
    amanasifkhalid
    on Feb 25, 2025
  9. JulieLeeMSFT
    removed
    untriagedNew issue has not been triaged by the area owner
    on Feb 25, 2025
  10. JulieLeeMSFT
    added this to the 10.0.0 milestone on Feb 25, 2025

  11. JulieLeeMSFT commented on Feb 25, 2025

    @JulieLeeMSFT
    JulieLeeMSFT
    on Feb 25, 2025
    Member

    @amanasifkhalid, PTAL.
    CC @jkoritzinsky.

  12. amanasifkhalid commented on Feb 25, 2025

    @amanasifkhalid
    amanasifkhalid
    on Feb 25, 2025
    Contributor

    some of them are meant to be published every time. What stopped producing them/marking them as shipping?

    Are there any that shouldn't be shipped for servicing?

    I suspect if you want them to build with every release you might need to determine if that would have any side-effects - make sure the packages are setup to flow / version correctly / etc.

    Aside from kicking off an official build, is there anything else we can/should do to verify these packages can be shipped before the next servicing release?

  13. jkotas commented on Feb 25, 2025

    @jkotas
    jkotas
    on Feb 25, 2025
    Member

    Are there any that shouldn't be shipped for servicing?

    The ones that ship as part of GA should be shipped as servicing too.

    Number of packages in this directory are non shipping, for example, https://github.com/dotnet/runtime/blob/main/src/coreclr/.nuget/ILCompiler.Reflection.ReadyToRun.Experimental/ILCompiler.Reflection.ReadyToRun.Experimental.pkgproj#L5 . This package does not ship as part of GA and it should not be shipped during serving either.

  14. amanasifkhalid commented on Feb 25, 2025

    @amanasifkhalid
    amanasifkhalid
    on Feb 25, 2025
    Contributor

    The ones that ship as part of GA should be shipped as servicing too.

    Thanks -- I suppose we should start by removing the catch-all rule here, and let the package configs handle themselves. Would getting a successful official build with this change on top of a copy of the release/9.0-staging branch be a good starting point for verifying this change? Or do servicing builds use a different process?

  15. JulieLeeMSFT commented on Feb 26, 2025

    @JulieLeeMSFT
    JulieLeeMSFT
    on Feb 26, 2025
    Member

    We think there is no different process for servicing builds.
    CC @carlossanlop.

  16. amanasifkhalid commented on Feb 27, 2025

    @amanasifkhalid
    amanasifkhalid
    on Feb 27, 2025
    Contributor

    I ran an official build of release/9.0-staging with the blanket rule removed from src/coreclr/.nuget/Directory.Build.props, and I can verify the same CoreCLR packages available in GA are under PackageArtifacts.

    Number of packages in this directory are non shipping, for example, https://github.com/dotnet/runtime/blob/main/src/coreclr/.nuget/ILCompiler.Reflection.ReadyToRun.Experimental/ILCompiler.Reflection.ReadyToRun.Experimental.pkgproj#L5 . This package does not ship as part of GA and it should not be shipped during serving either.

    I'm finding this package available under PackageArtifacts in every official non-servicing build (example) -- I'm guessing this shouldn't be the case?

  17. ViktorHofer commented on Feb 27, 2025

    @ViktorHofer
    ViktorHofer
    on Feb 27, 2025
    Member

    The PackageArtifacts folder contains both the non-shipping and shipping packages. The non-shipping ones go to a transport feed. That's intentional. You should find a MergedManifest.xml under BlobArtifacts which should tell you which packages are marked as shipping.

  18. amanasifkhalid commented on Feb 27, 2025

    @amanasifkhalid
    amanasifkhalid
    on Feb 27, 2025
    Contributor

    Thanks @ViktorHofer! The shipping statuses in the MergedManifest.xml from my official build look good:

    <Package Id="dotnet-pgo" Version="9.0.3-servicing.25126.13" DotNetReleaseShipping="false" NonShipping="true"/>
<Package Id="ILCompiler.Reflection.ReadyToRun.Experimental" Version="9.0.3-servicing.25126.13" DotNetReleaseShipping="false" NonShipping="true"/>
...
<Package Id="Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>

    Here's what the manifest looks like for an earlier servicing build:

    <Package Id="dotnet-pgo" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>
<Package Id="ILCompiler.Reflection.ReadyToRun.Experimental" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>
...
<Package Id="Microsoft.NETCore.ILAsm" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>
<Package Id="Microsoft.NETCore.ILDAsm" Version="9.0.3-servicing.25125.6" DotNetReleaseShipping="false" NonShipping="true"/>

    I'm guessing we want the version suffix for ILAsm/ILDasm set as well. I'll fix that.

  19. amanasifkhalid commented on Feb 27, 2025

    @amanasifkhalid
    amanasifkhalid
    on Feb 27, 2025
    Contributor

    I'm guessing we want the version suffix for ILAsm/ILDasm set as well. I'll fix that.

    After messing with this locally, I'm guessing the desired package version is passed during the build (i.e. /p:PackageVersion=x.y.z). Is it sufficient to just allow the StableVersion of shipped servicing packages to be set here in src\coreclr\.nuget\Directory.Build.targets\? With that approach, the version suffix for shipped packages is 9.0.3 in my local builds. For an actual servicing release, would we manually specify the package version anyway?

  20. ericstj commented on Feb 27, 2025

    @ericstj
    ericstj
    on Feb 27, 2025
    MemberAuthor

    I did an audit of all the packages that ship in ga and which were missing from servicing and found the complete list here: https://gist.github.com/ericstj/2be44b58fab7026875164a5e66020910

    Have a review in case that includes any other packages owned by area-Infrastructure-coreclr

    For an actual servicing release, would we manually specify the package version anyway?

    That really depends on what you want the behavior to be. If you want them to always ship and follow the product versioning (preferred) then you shouldn't specify any versions manually.
    If you instead wanted to selectively enable and disable, then you'd want to specify the version independently.

    At one point long ago in coreclr branch these packages built using the latter policy. I think you can stop doing that since folks forgot how to do it anyways. Ship all the time. Really the only downside of doing that is if your downstream consumers understand the regular updates and they work well for them. If those consumers are part of the product for instance - make sure they haven't done something like hardcode Major.0.0.

  21. amanasifkhalid commented on Feb 28, 2025

    @amanasifkhalid
    amanasifkhalid
    on Feb 28, 2025
    Contributor

    Thanks @ericstj for putting together a list! I think we have all the CoreCLR packages covered. From the build manifest:

    <Package Id="dotnet-ilverify" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.ILVerification" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="Microsoft.NET.Sdk.IL" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-musl-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.linux-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.osx-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-arm64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-arm64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x64.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x64.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x86.Microsoft.NETCore.ILAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>
<Package Id="runtime.win-x86.Microsoft.NETCore.ILDAsm" Version="9.0.3" DotNetReleaseShipping="true" NonShipping="false"/>

    For that particular build, the IsStable property is false -- is that why these packages all lack version suffixes? I suspect for stable builds, the correct suffix will be appended, such as in this build.

  22. ViktorHofer commented on Feb 28, 2025

    @ViktorHofer
    ViktorHofer
    on Feb 28, 2025
    Member

    For that particular build, the IsStable property is false -- is that why these packages all lack version suffixes? I suspect for stable builds, the correct suffix will be appended.

    That's the expectation, yes. We should double check when we do the next servicing release that these have indeed the correct version suffix. Or we queue a test stable build - unsure if that's worth the hassle.

    We should follow-up on the remaining non-runtime packages (sourcelink, vs-codecoverage and llvm-project). Ignore test-templates as that repo got merged into sdk and is unlikely to change in the active servicing branches.

    For sourcelink, I don't think anything in the repo infrastructure prevents the packages from getting built during servicing. This might be a case where the release infra team intentionally doesn't flow updates during servicing. cc @mmitche @leecow

  23. dotnet-policy-service
    added
    in-prThere is an active PR which will close this issue when it is merged
    on Feb 28, 2025
  24. amanasifkhalid
    closed this as completedin #113020on Feb 28, 2025
  25. amanasifkhalid
    added a commit that references this issue on Feb 28, 2025

    Ship CoreCLR packages in servicing releases (#113020)

    bb5f473
  26. github-project-automation
    moved this to Done in Runtime Infraon Feb 28, 2025
  27. github-actions
    locked and limited conversation to collaborators on Mar 31, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area-Infrastructure-coreclrOnly use for closed issuesin-prThere is an active PR which will close this issue when it is merged

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions