Unable to build and run aspnetcore unit tests on RHEL 8/arm64

[omajid]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

I am trying to build and run ASP.NET Core repo and its tests on RHEL 8 + arm64, a supported platform. It fails:

$ git clone https://github.com/dotnet/aspnetcore .git
$ cd aspnetcore
$ git checkout main
Already on 'main'
Your branch is up to date with 'origin/main'.
$ git rev-parse HEAD
06155c05af89c957de20d2c53cee0e37171b9a09
$ git submodule init
Submodule 'src/submodules/MessagePack-CSharp' (https://github.com/aspnet/MessagePack-CSharp.git) registered for path 'src/submodules/MessagePack-CSharp'
Submodule 'src/submodules/googletest' (https://github.com/google/googletest) registered for path 'src/submodules/googletest'
$ git submodule update --recursive
$ eng/build.sh --ci --arch arm64 -c Release /p:IsPlaywrightAvailable=false --restore --build --pack
warning: No default group of projects was specified, so building the 'managed' and its dependent subset of projects. Run build.sh --help for more details.
warning: Building of C# project is enabled and has dependencies on NodeJS projects. Building of NodeJS projects is enabled since node is detected on PATH.
warning: Note that if you are running Source Build, building NodeJS projects will be disabled later on.
##vso[task.setvariable variable=Artifacts;isSecret=false;isOutput=true]/home/tester/aspnetcore/artifacts
##vso[task.setvariable variable=Artifacts.Toolset;isSecret=false;isOutput=true]/home/tester/aspnetcore/artifacts/toolset
##vso[task.setvariable variable=Artifacts.Log;isSecret=false;isOutput=true]/home/tester/aspnetcore/artifacts/log/Release
##vso[task.setvariable variable=Temp;isSecret=false;isOutput=true]/home/tester/aspnetcore/artifacts/tmp/Release
##vso[task.setvariable variable=TMP;isSecret=false;isOutput=true]/home/tester/aspnetcore/artifacts/tmp/Release
Downloading 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'
...
  Restored /home/tester/aspnetcore/src/Caching/SqlServer/src/Microsoft.Extensions.Caching.SqlServer.csproj (in 574 ms).
  Restored /home/tester/aspnetcore/src/Framework/Microsoft.Internal.Aspnetcore.DotNetApiDocs.Transport/src/Microsoft.Internal.Aspnetcore.DotNetApiDocs.Transport.csproj (in 1.47 sec).
  Microsoft.AspNetCore.Grpc.JsonTranscoding -> /home/tester/aspnetcore/artifacts/bin/Microsoft.AspNetCore.Grpc.JsonTranscoding/Release/net9.0/Microsoft.AspNetCore.Grpc.JsonTranscoding.dll
/home/tester/aspnetcore/.packages/grpc.tools/2.64.0/tools/linux_arm64/protoc : error : /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /home/tester/aspnetcore/.packages/grpc.tools/2.64.0/tools/linux_arm64/protoc) [/home/tester/aspnetcore/src/Grpc/JsonTranscoding/perf/Microsoft.AspNetCore.Grpc.Microbenchmarks/Microsoft.AspNetCore.Grpc.Microbenchmarks.csproj]
##vso[task.logissue type=error;sourcepath=/home/tester/aspnetcore/.packages/grpc.tools/2.64.0/tools/linux_arm64/protoc;linenumber=0;columnnumber=0;code=;](NETCORE_ENGINEERING_TELEMETRY=Build) /lib64/libm.so.6: version `GLIBC_2.29' not found (required by /home/tester/aspnetcore/.packages/grpc.tools/2.64.0/tools/linux_arm64/protoc)

Please note that this is the aspnetcore repo in isolation, outside of the VMR/source-build context.

Expected Behavior

I can build and run ASP.NET Core repo and its unit tests on RHEL 8.

.NET 9 lists RHEL 8 + arm64's glibc as suppported: https://github.com/dotnet/core/blob/main/release-notes/9.0/supported-os.md#linux-compatibility. Does/should that include being able to build aspnetcore and run its unit tests on that platform?

Steps To Reproduce

On a RHEL 8 + arm64 system, run the following commands in a aspnetcore repo checkout:

eng/build.sh --ci --arch arm64 -c Release /p:IsPlaywrightAvailable=false --restore --build --pack

Exceptions (if any)

No response

.NET Version

bootstrap SDK is 9.0.100-preview.7.24371.4

Anything else?

Issue against grpc.tools: grpc/grpc#37340

[omajid]

.NET 9 lists RHEL 8 + arm64's glibc as suppported. Does/should that include being able to build aspnetcore and run its unit tests on that platform?

@leecow @MichaelSimons @richlander @wtgodbe

[amcasey]

Tagging @JamesNK for grpc.tools.

[richlander]

The support matrix is for the whole product. If there are differences, those should be clearly documented.

Ideally, all of our dependencies satisfy the same support level as the product.

[JamesNK]

/lib64/libm.so.6: version `GLIBC_2.29' not found (required by /home/tester/aspnetcore/.packages/grpc.tools/2.64.0/tools/linux_arm64/protoc)

It looks like Grpc.Tools requires a dependency that's not installed?

[omajid]

It looks like Grpc.Tools requires a dependency that's not installed?

Yes. Grpc.Tools requires new versions of core libraries (like libm.so) that are are a part of glibc. In other words, it needs a newer glibc. glibc is the main C library. Almost everything links against it. If it is not installed, most applications (eg, bash) in common Linux distros will work. Upgrading glibc is not generally an option: it is so invasive and risky that it's probably a better idea to upgrade to a newer distro instead.

[JamesNK]

Is this a new problem? Grpc.Tools was recently updated to a newer version. Perhaps the newer version uses a newer glibc.

Grpc.Tools is only used in tests and in the gRPC templates. It doesn't prevent the rest of ASP.NET Core from working (although it does break the build if they're built together)

I don't know what the workaround would be other than gRPC from downgrading Grpc.Tools and never updating it until RHEL 9 is no longer supported. But that doesn't seem workable.

[omajid]

Is this a new problem?

Looks like it. I have some internal CI runs on RHEL 8+ arm64 from June and July and those were working.

I don't know what the workaround would be other than gRPC from downgrading Grpc.Tools and never updating it until RHEL 9 is no longer supported. But that doesn't seem workable.

Agreed, that's not sustainable. I think this need to be fixed in grpc itself in the long term. I reported this issue to grpc repo as well (grpc/grpc#37340) and we may even have a fix.

[wtgodbe]

Marking this as blocked until we hear back on grpc/grpc#37340