ModuleNotFoundError: No module named 'pkg_resources'

[jkotas]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=910866

Build leg reported

ComInterfaceGenerator.Tests.WorkItemExecution

Pull Request

dotnet/runtime#110558

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "ModuleNotFoundError: No module named 'pkg_resources'",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=910866
Error message validated: [ModuleNotFoundError: No module named 'pkg_resources']
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/8/2025 10:20:43 PM UTC

Report

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	0

[dougbu]

we're tracking work on this in dotnet/runtime#4751. we found the problem another way and noticed it seems to be specific to ArmArch Linux Docker containers just yesterday. hoping to move away from the deprecated and, now, apparently sometimes unavailable package in time to include the fix in our next rollout

am I correct the containers failing in your build are quite infrequently used❓

[jkotas]

This is failing on many PRs in dotnet/runtime. The affected containers are used by default dotnet/runtime CI configuration.

[jkotas]

This is failing on many PRs in dotnet/runtime

You can see it in the stats in the top post.

[jkotas]

it seems to be specific to ArmArch Linux

This affects number of Linux and macOS variants.

For example, here is a log from macOS x64: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/Microsoft.Extensions.Configuration.Tests/1/console.47b8555d.log?helixlogtype=result

[akoeplinger]

I think the common factor is having Python 3.12, e.g. if you look at https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/ComInterfaceGenerator.Tests/1/console.aa96da09.log?helixlogtype=result from the same job as Jan posted above it worked, because the dci-mac-build-133 macOS machine is not using Python 3.12 (I guess because it wasn't updated yet)

[dougbu]

I think the common factor is having Python 3.12, e.g. if you look at https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/ComInterfaceGenerator.Tests/1/console.aa96da09.log?helixlogtype=result from the same job as Jan posted above it worked, because the dci-mac-build-133 macOS machine is not using Python 3.12 (I guess because it wasn't updated yet)

I agree. it also seems like the problem is only partially under our direct control. we use pkg_resources in a couple of places but so does the azure package version we rely on. other dependencies seem to have the right try / catch setup to fallback when pkg_resources isn't available.

---

unless we missed some older builds that failed, problems started just after our rollout began updating queues yesterday. the earliest error I saw had timestamp 2025-01-08T21:20:05.2091700Z and the queue rollout started real work at 2025-01-08T20:45:17.1221544Z (slightly late due to retries of earlier jobs in our pipeline). the rollout picked up new OS packages for Python3 on Linux machines as well as slightly changed Python requirements for our helix-scripts/ code. the second part might have impacted OSX machines

[dougbu]

Python 3.12.8 released on 2024-12-03. it contains a bunch of fixes and dependency updates though nothing obviously linked to pkg_resources. it did upgrade its bundled pip to 24.3.1 but the pip changelog also doesn't include an obvious smoking gun. it also upgraded its libexpat dependency to 2.6.3 but that's written in C

I checked the setuptools changelog as well b/c we let that float as much as the Python and pip versions allow. nothing obvious there either

[dougbu]

we're reverting yesterday's rollout due to the problems discussed in this issue. queues are getting updated as I type this note

[dougbu]

revert is now complete but I don't see a clear signal that this particular problem has been resolved. please let us know