HTTPS requests fail on Ubuntu 24.04 Noble ARM32 due to bundled certs "NotTimeValid" error

[lbussell]

Update: A final resolution to this issue has been posted at dotnet/core#9285.

Description

HTTPS requests from .NET are failing on Ubuntu 24.04 Noble on ARM32 due to bundled certs "NotTimeValid" errors.

I believe this may be because 24.04 has migrated to 64-bit time. The OpenSSL package has changed from libssl3 to libssl3t64.

Reproduction Steps

1. dotnet new console
2. dotnet add package System.Text.Json
3. .NET will try to restore the package from NuGet.org and fail with the error under "Actual behavior" below.

I created this repro Dockerfile. It may or may not work since I had issues with it on my AMD64 dev machine. No easy way for me to test this currently outside of our official .NET Container image builds. With some trial and error you can probably get the issue to repro on a real arm32 machine using Docker. https://gist.github.com/lbussell/52e0ac904108d238d0e511f8b6ec89e1

Expected behavior

The .NET CLI should successfully access NuGet.org.

Actual behavior

From our PR validation in .NET Docker: https://dev.azure.com/dnceng-public/public/_build/results?buildId=652911&view=logs&j=7bc65791-3246-5ca2-874f-59d2e579cf6b&t=08651ed6-ba3f-5f8a-52df-50083ac157c2&l=942

 #10 2.407 info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.204/trustedroots/codesignctl.pem'.
 #10 2.491 info : X.509 certificate chain validation will use the fallback certificate bundle at '/usr/share/dotnet/sdk/8.0.204/trustedroots/timestampctl.pem'.
 #10 2.494 info : Adding PackageReference for package 'foo' into project '/source/app/app.csproj'.
 #10 8.602 error: Unable to load the service index for source https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet6/nuget/v3/index.json.
 #10 8.602 error:   The SSL connection could not be established, see inner exception.
 #10 8.602 error:   The remote certificate is invalid because of errors in the certificate chain: NotTimeValid

Regression?

No response

Known Workarounds

No response

Configuration

• .NET Runtime: 8.0.4 and 9.0.0-preview.4.24218.7
• OS: Ubuntu 24.04 "Noble"
• Arch: ARM
• Do you know whether it is specific to that configuration? Yes it is specific to this config. Likely affects older .NET versions as well.

Other information

Discovered in dotnet/dotnet-docker#5241

Known build error template

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=674655&view=results
Build error leg or test failing:
Pull request:

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["arm32v7", "The remote certificate is invalid because of errors in the certificate chain: NotTimeValid"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=674655
Error message validated: [arm32v7 The remote certificate is invalid because of errors in the certificate chain: NotTimeValid]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 5/16/2024 3:23:33 PM UTC

Report

Build	Definition	Test	Pull Request
674655	dotnet/runtime	System.Net.WebSockets.Client.Tests.ArraySegmentSendReceiveTest.ReceiveAsync_MultipleOutstandingReceiveOperations_Throws

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	1

[vcsjones]

I can take a look (and finally have an excuse to take that Raspberry Pi out of the box!)

[vcsjones]

Firstly

1. I can reproduce it

   Distributor ID:	Ubuntu
   Description:	Ubuntu Noble Numbat (development branch)
   Release:	24.04
   Codename:	noble
   root@1dc47c31f061:/project# dotnet add package System.Text.Json
     Determining projects to restore...
     Writing /tmp/tmpiHt9zR.tmp
   info : X.509 certificate chain validation will use the fallback certificate bundle at '/root/dotnet/sdk/8.0.204/trustedroots/codesignctl.pem'.
   info : X.509 certificate chain validation will use the fallback certificate bundle at '/root/dotnet/sdk/8.0.204/trustedroots/timestampctl.pem'.
   info : Adding PackageReference for package 'System.Text.Json' into project '/project/project.csproj'.
   error: Unable to load the service index for source https://api.nuget.org/v3/index.json.
   error:   The SSL connection could not be established, see inner exception.
   error:   The remote certificate is invalid because of errors in the certificate chain: NotTimeValid
   

2. .NET "knows" it is the right time. So it's not like the clock is wrong, or that the source of time is wrong.

   root@1dc47c31f061:/project# date
   Tue Apr 23 21:40:27 EDT 2024
   root@1dc47c31f061:/project# dotnet run
   Hello, World!04/23/2024 21:40:37
   root@1dc47c31f061:/project#
   

3. OpenSSL CLI is okay with the host. So, it's not like the certificate really is expired or there is some underlying problem in OpenSSL.

   No client certificate CA names sent
   Peer signing digest: SHA256
   Peer signature type: RSA-PSS
   Server Temp Key: ECDH, prime256v1, 256 bits
   ---
   SSL handshake has read 4273 bytes and written 441 bytes
   Verification: OK
   ---
   

4. The X509Certificate2 is able to load the notBefore and notAfter correctly. (.NET and OpenSSL differ by my timezone offset since .NET is showing it in local time and OpenSSL is showing it in UTC time)

   root@d2b82c600199:/project# dotnet run
   10/18/2023 16:00:08
   10/12/2024 16:00:08
   Hello, World!
   root@d2b82c600199:/project# openssl x509 -in cert.pem -text -noout
   Certificate:
       Data:
           Version: 3 (0x2)
           Serial Number:
               33:00:0d:53:71:cb:65:3d:f3:ee:7c:dc:5c:00:00:00:0d:53:71
           Signature Algorithm: sha384WithRSAEncryption
           Issuer: C = US, O = Microsoft Corporation, CN = Microsoft Azure RSA TLS Issuing CA 07
           Validity
               Not Before: Oct 18 20:00:08 2023 GMT
               Not After : Oct 12 20:00:08 2024 GMT
   

This will take a bit of additional time to troubleshoot, whatever is going on is not immediately obvious (to me)

[dviererbe]

@vcsjones libssl3t64 changed the ABI compared to libssl3. This may cause the error.

[vcsjones]

Yep. I am trying to narrow it down to the smallest possible place to start looking for ABI mismatches.

It does reproduce with X509Chain.Build. So it is something specifically with chain building. Reading the dates off the individual certificates seems okay.

Right now I believe this code to be problematic under the ABI change.

runtime/src/native/libs/System.Security.Cryptography.Native/openssl.c

Lines 953 to 968 in 5661009

	time_t verifyTime = MakeTimeT(year, month, day, hour, minute, second, isDst);
	if (verifyTime == (time_t)-1)
	{
	return 0;
	}
	X509_VERIFY_PARAM* verifyParams = X509_STORE_get0_param(ctx);
	if (!verifyParams)
	{
	return 0;
	}
	X509_VERIFY_PARAM_set_time(verifyParams, verifyTime);
	return 1;

We always set the verification time when building the chain:

runtime/src/libraries/System.Security.Cryptography/src/System/Security/Cryptography/X509Certificates/OpenSslX509ChainProcessor.cs

Line 120 in 5661009

Interop.Crypto.X509StoreSetVerifyTime(store, verificationTime);

Next steps is I need to figure out how to get a debugger going here.

[bartonjs]

@AndyAyersMS On ARM32, if time_t is a 64-bit value then it gets passed using the stack pointer offset convention rather than as a register, right? That suggests to me that we need to build/distribute an entirely different runtime for arm32+time_t64 and arm32+time_t32.

Unless our call to X509_VERIFY_PARAM_set_time is literally the only place in all of .NET that a time_t is passed by value, in which case I think we can use the time(2) function to detect the size of time_t at runtime:

uint64_t timeOrBigger = 0xFFFFFFFFFFFFFFFF;
time((time_t*)&timeOrBigger);

if (timeOrBigger >> 32 == 0xFFFFFFFF)
{
    // Either a: the top 32 bits didn't get written, or
    // b: it's approximately the year 128,000,002,000 and all times start with that pattern.
    //
    // Assume a, and be happy
    // normal mktime, normal call.
}
else
{
    uint64_t (*mktime_ptr)(struct tm*) = (whatever the cast looks like)(void*)mktime;
    uint64_t timeval = mktime_ptr(&tm);
    
    void (*hacky)(X509_VERIFY_PARAM*, uint64_t) = (cast)(void*)X509_VERIFY_PARAM_set_time;
    hacky(verifyParams, timeval);

}

But I highly suspect that this isn't the only thing broken right now... just the one people see first. (And I'm concerned that it means any function called that wants a time_t is going to be reading arbitrary stack state from the caller, which seems... bad).

[bartonjs]

runtime/src/coreclr/pal/src/file/filetime.cpp

Lines 229 to 265 in ce0ae49

	BOOL PALAPI FileTimeToSystemTime( CONST FILETIME * lpFileTime,
	LPSYSTEMTIME lpSystemTime )
	{
	UINT64 FileTime = 0;
	time_t UnixFileTime = 0;
	struct tm * UnixSystemTime = 0;
	/* Combine the file time. */
	FileTime = lpFileTime->dwHighDateTime;
	FileTime <<= 32;
	FileTime |= (UINT)lpFileTime->dwLowDateTime;
	bool isSafe = ClrSafeInt<UINT64>::subtraction(
	FileTime,
	SECS_BETWEEN_1601_AND_1970_EPOCHS * SECS_TO_100NS,
	FileTime);
	if (isSafe == true)
	{
	#if HAVE_GMTIME_R
	struct tm timeBuf;
	#endif /* HAVE_GMTIME_R */
	/* Convert file time to unix time. */
	if (((INT64)FileTime) < 0)
	{
	UnixFileTime = -1 - ( ( -FileTime - 1 ) / 10000000 );
	}
	else
	{
	UnixFileTime = FileTime / 10000000;
	}
	/* Convert unix file time to Unix System time. */
	#if HAVE_GMTIME_R
	UnixSystemTime = gmtime_r( &UnixFileTime, &timeBuf );
	#else /* HAVE_GMTIME_R */
	UnixSystemTime = gmtime( &UnixFileTime );
	#endif /* HAVE_GMTIME_R */

sends a time_t* into gmtime/gmtime_r, which means (assuming this file is used on arm32) it's getting the 32 bits we assigned, and the 32 bits declared after (before?) it on the stack. Which might be our conveniently zero value for UnixSystemTime, or might be part of FileTime, or might just be garbage.