CryptographicException in NonPowerOfTwoKeySizeOaepRoundtrip test

[MihaZupan]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1170333
Build error leg or test failing: System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.NonPowerOfTwoKeySizeOaepRoundtrip
Pull request: #119673

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["NonPowerOfTwoKeySizeOaepRoundtrip", "CryptographicException"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.NonPowerOfTwoKeySizeOaepRoundtrip(oaepPaddingMode: OaepSHA3-512) [FAIL]
      System.Security.Cryptography.CryptographicException : Error occurred during a cryptographic operation.
      Stack Trace:
        /_/src/libraries/Common/src/Interop/Unix/System.Security.Cryptography.Native/Interop.EvpPkey.Rsa.cs(46,0): at Interop.Crypto.RsaGenerateKey(Int32 keySize)
        /_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs(724,0): at System.Security.Cryptography.RSAOpenSsl.GenerateKey()
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(327,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ViaFactory(LazyThreadSafetyMode mode)
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(345,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor)
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(425,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].CreateValue()
        /_/src/libraries/System.Private.CoreLib/src/System/Lazy.cs(505,0): at System.Lazy`1[[System.Security.Cryptography.SafeEvpPKeyHandle, System.Security.Cryptography, Version=10.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a]].get_Value()
        /_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs(712,0): at System.Security.Cryptography.RSAOpenSsl.GetKey()
        /_/src/libraries/Common/src/System/Security/Cryptography/RSAOpenSsl.cs(236,0): at System.Security.Cryptography.RSAOpenSsl.TryEncrypt(ReadOnlySpan`1 data, Span`1 destination, RSAEncryptionPadding padding, Int32& bytesWritten)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(52,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.<>c__DisplayClass0_0.<Encrypt>b__0(Byte[] dest)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(62,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.TryWithOutputArray(Func`2 func)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.netcoreapp.cs(52,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.Encrypt(RSA rsa, Byte[] data, RSAEncryptionPadding padding)
        /_/src/libraries/Common/tests/System/Security/Cryptography/AlgorithmImplementations/RSA/EncryptDecrypt.cs(701,0): at System.Security.Cryptography.Rsa.Tests.EncryptDecrypt.NonPowerOfTwoKeySizeOaepRoundtrip(RSAEncryptionPadding oaepPaddingMode)
           at System.Object.InvokeStub_EncryptDecrypt.NonPowerOfTwoKeySizeOaepRoundtrip(Object , Span`1 )
        /_/src/libraries/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.cs(95,0): at System.Reflection.MethodBaseInvoker.InvokeWithOneArg(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1170333
Error message validated: [at Interop.Crypto.RsaGenerateKey Error occurred during a cryptographic operation]
Result validation: ❌ Known issue did not match with the provided build.
Validation performed at: 10/10/2025 6:00:46 PM UTC

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1170333
Error message validated: [NonPowerOfTwoKeySizeOaepRoundtrip CryptographicException]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 10/10/2025 6:23:48 PM UTC

Report

Build	Definition	Test	Pull Request
1184824	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#120519
1184241	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution
1181329	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#120944
1177287	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#120822
1176844	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#120769
1175804	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#120769
1172121	dotnet/runtime	System.Security.Cryptography.Tests.WorkItemExecution	#120648
1170333	dotnet/runtime	System.Security.Cryptography.Rsa.Tests.EncryptDecrypt_TrySpan.NonPowerOfTwoKeySizeOaepRoundtrip	#119673

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	2	8

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[MihuBot]

I'm a bot. Here is a possible related and/or duplicate issue (I may be wrong):

• STATUS_UNSUCCESSFUL in RsaCryptRoundtrip_OaepSHA1 #29683