[Android] Normalize RSA no-padding output to modulus size to fix VerifyHash

[simonrozsival]

It appears some versions of the Android crypto libraries the "RSA/ECB/NoPadding" returns results as minimal-length big-endian integers (leading 0x00 trimmed), causing RsaVerificationPrimitive to return fewer bytes than the modulus (e.g., 47 vs 48). This breaks RSAAndroid.VerifyHash and tests like PublicKeyTests.TestKey_RSA384_ValidatesSignature fail.

In the failing test, the decryptedBytes contained the following 47 bytes when running on a Samsung device with Android API 35:

0x01 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0x00 0x30 0x21 0x30 0x09 0x06 0x05 0x2B 0x0E 0x03 0x02 0x1A 0x05 0x00 0x04 0x14 0xF7 0xFF 0x9E 0x8B 0x7B 0xB2 0xE0 0x9B 0x70 0x93 0x5A 0x5D 0x78 0x5E 0x0C 0xC5 0xD9 0xD0 0xAB 0xF0

The leading 0x00 byte in the signature padding is missing in this case (the expected RSA signature padding is 0x00 0x01 0xFF ... 0xFF 0x00).

Fixes #118366

/cc @bartonjs @vcsjones @vitek-karas @BrzVlad

[simonrozsival]

/azp run runtime-extra-platforms

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[Copilot]

Pull Request Overview

This PR fixes a compatibility issue with Android crypto libraries where RSA verification operations return minimal-length results instead of modulus-sized outputs. The fix ensures that RSA verification results are properly normalized to the expected modulus size by left-padding with zeros when necessary.

Key changes:

• Added validation to ensure decrypted bytes don't exceed expected length
• Implemented left-padding with zeros to normalize output to modulus size
• Updated return value calculation to reflect the normalized length

[simonrozsival]

/backport to release/10.0-rc1

[github-actions]

Started backporting to release/10.0-rc1: https://github.com/dotnet/runtime/actions/runs/17096489581

[vcsjones]

@simonrozsival does this impact Android on net9.0 on newer Android versions? If so it might be worth taking the backport back to 9 and 8.

[simonrozsival]

@vcsjones yes, we should backport this to .NET 9 as well. I don't know if we should backport to .NET 8 as well, since .NET MAUI 8 is already out of support.

[simonrozsival]

/backport to release/9.0-staging

[github-actions]

Started backporting to release/9.0-staging: https://github.com/dotnet/runtime/actions/runs/17099292564

[akoeplinger]

I don't know if we should backport to .NET 8 as well, since .NET MAUI 8 is already out of support.

Given #118366 happened in CI I'd say we should at least backport disabling the test if we don't backport the fix. Or disable running Android jobs in 8.0 😄

[simonrozsival]

/backport to release/10.0

[github-actions]

Started backporting to release/10.0: https://github.com/dotnet/runtime/actions/runs/17120602273