segmentation fault with rsa-pss when pss params don't match the operation

[panva]

Version

v16.7.0

Platform

Darwin C02CX0K5MD6V 20.6.0 Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:31 PDT 2021; root:xnu-7195.141.2~5/RELEASE_X86_64 x86_64

Subsystem

crypto

What steps will reproduce the bug?

const crypto = require('node:crypto')

const pk = `-----BEGIN PRIVATE KEY-----
MIIE7wIBADA9BgkqhkiG9w0BAQowMKANMAsGCWCGSAFlAwQCAaEaMBgGCSqGSIb3
DQEBCDALBglghkgBZQMEAgGiAwIBEASCBKkwggSlAgEAAoIBAQDfqNM4C+QtD73i
ILqOkqfV8ha3O19jpX8UujIk1Z72bbbuwEzh0+sBw0dD0N8CgkXnePOEEd6q7HNm
byCNqRpDK6NDvaCMDWgEaD/PlHkRntvKh81IXSMC5imjRfOcZIE/Gnw7h8tanab0
n75+ODvLJrmEWUG2q79Im1mWMx7Spod+Np6XEY+7I7nAUUWivr35Yx5DeyxY8rxF
GpsLtGsi7JNQO4aHyeBpj8tz0Fhv23uPywE2nGmPHfnkXWbrTcHGbzYBgEbeSH9K
UkRwczqDXNOPhtfaEHEFTm0MoeKCnJe1VOjSywev77dV1KZfpVh3Kh0ZRQIe9YOV
Jhj4lMx3AgMBAAECggEBAIc+IgK5Bg/NfgeXvNdrjPuM+PlxeHvb3h1dfebSGd5v
d3elZpgDug6F07kJO2Db/4M5mx7YY2m9swZU2j1u7MeDQqU6rDMkBCruEu/lmtPx
2Hv+ZD6Gux4MqU7mhKmkCJds34Rr16aCwCsZ0WmnfViZoQKLqnXYIsG31pNBdDjx
gke0HhX1LkA9yTVwlk8xOaHPqI4KfsFAyoiiHzyttGDexzb1PzmM0pybAPDMhpN/
wXp2kLjyzmUmPe2Y2yva69WVWo7qS6joKjY75MQ1t20HYgEL69IApvCPu4CANfi9
j3FAaV/+WrnhKCi6QyUi5PCI/+AJLsjNQmqTXIdBEoECgYEA+XsgFbeZ6+ZzEHa7
HyFH6kiyBLd0q7w+ZLPsoOmEApDaP3yXSC7eJU7M/tPUPj8VQMMSK2D6fgmUDwhb
3mEXFZxf67UlPFsFjweYcBihwy4r8QKBwury6dEbHPSUq4mXFJF5XRQdGqRGkr/F
8OLZ0MwmHLUzczA67PxP/wF8TsECgYEA5YD4RxxJJYfAk1rFbqHRhNB8UeYVL+Wo
wsRET1JeFg+S5grLGUga+KBB8Jn7Ahaip7MVE0Iud1cgaDi4WBEJsbJ6oJTlHJEg
Jq7QAaBafwjeSCSnaEsVBVNvriy2WF7uAomLSKmW6uSUOBBFFt4+G+akG56EfOPc
7YKBfsf5ITcCgYBvjVZzX307tfeNTQmuibsWTxsKcN2CTNG5RZpw+PlGDG8KJDOg
2xQJqoqPBzjH/H0MUC03qE1ZPf8uGZa6gL9JsnpRctYLfselhMfsl5b9JxAO3AgZ
l+S2GAH/mH1BlmwvjjyuGehJmVrVE1r2sviiHCaOf5dZ0h8HCGrco1VqAQKBgQCf
fYkMofOTSUvjG2mpAHuCOQCsSaDfsFIfSBXQqgUIf7ouc8HAyAM2VOh+NAPj56cR
s7opsAxqkvnKc+BoEy8Rdl8RyWeO+qvFNicHelBph9gxeod8SvFIyjsKZ7gwoYf1
63AIBxMCGeeHLodU5Q10hkv1hau8vv2BcPhdCstu8QKBgQDgO4Rr36Pa5rjbNbGN
PsJqALjthTeU0yaU8hdC7Hc7B/T6npEIhw3s1eNq7e0eeDltqz7tDnY8qysazbQZ
p1cV5TJ8+gtwGmDoADBnU1NXqX4Squfml6OhNEucpTdjux7JdLVmmQFraOT2Eu5f
9uuNtA+d8uhBEXhskuvEC552ug==
-----END PRIVATE KEY-----`

crypto.sign('sha256', Buffer.from('hello'), pk) // works
crypto.sign('sha384', Buffer.from('hello'), pk) // segmentation fault

How often does it reproduce? Is there a required condition?

always

What is the expected behavior?

crypto.sign and crypto.verify would throw an appropriate error

What do you see instead?

segmentation fault

Additional information

The private key is taken directly from /test/fixtures/keys/rsa_pss_private_2048_sha256_sha256_16.pem

[panva]

SubtleCrypto.sign has the same issue

const crypto = require('node:crypto')
const { webcrypto: { subtle } } = crypto

const pk = Buffer.from(`MIIE7wIBADA9BgkqhkiG9w0BAQowMKANMAsGCWCGSAFlAwQCAaEaMBgGCSqGSIb3DQEBCDALBglghkgBZQMEAgGiAwIBEASCBKkwggSlAgEAAoIBAQDfqNM4C+QtD73iILqOkqfV8ha3O19jpX8UujIk1Z72bbbuwEzh0+sBw0dD0N8CgkXnePOEEd6q7HNmbyCNqRpDK6NDvaCMDWgEaD/PlHkRntvKh81IXSMC5imjRfOcZIE/Gnw7h8tanab0n75+ODvLJrmEWUG2q79Im1mWMx7Spod+Np6XEY+7I7nAUUWivr35Yx5DeyxY8rxFGpsLtGsi7JNQO4aHyeBpj8tz0Fhv23uPywE2nGmPHfnkXWbrTcHGbzYBgEbeSH9KUkRwczqDXNOPhtfaEHEFTm0MoeKCnJe1VOjSywev77dV1KZfpVh3Kh0ZRQIe9YOVJhj4lMx3AgMBAAECggEBAIc+IgK5Bg/NfgeXvNdrjPuM+PlxeHvb3h1dfebSGd5vd3elZpgDug6F07kJO2Db/4M5mx7YY2m9swZU2j1u7MeDQqU6rDMkBCruEu/lmtPx2Hv+ZD6Gux4MqU7mhKmkCJds34Rr16aCwCsZ0WmnfViZoQKLqnXYIsG31pNBdDjxgke0HhX1LkA9yTVwlk8xOaHPqI4KfsFAyoiiHzyttGDexzb1PzmM0pybAPDMhpN/wXp2kLjyzmUmPe2Y2yva69WVWo7qS6joKjY75MQ1t20HYgEL69IApvCPu4CANfi9j3FAaV/+WrnhKCi6QyUi5PCI/+AJLsjNQmqTXIdBEoECgYEA+XsgFbeZ6+ZzEHa7HyFH6kiyBLd0q7w+ZLPsoOmEApDaP3yXSC7eJU7M/tPUPj8VQMMSK2D6fgmUDwhb3mEXFZxf67UlPFsFjweYcBihwy4r8QKBwury6dEbHPSUq4mXFJF5XRQdGqRGkr/F8OLZ0MwmHLUzczA67PxP/wF8TsECgYEA5YD4RxxJJYfAk1rFbqHRhNB8UeYVL+WowsRET1JeFg+S5grLGUga+KBB8Jn7Ahaip7MVE0Iud1cgaDi4WBEJsbJ6oJTlHJEgJq7QAaBafwjeSCSnaEsVBVNvriy2WF7uAomLSKmW6uSUOBBFFt4+G+akG56EfOPc7YKBfsf5ITcCgYBvjVZzX307tfeNTQmuibsWTxsKcN2CTNG5RZpw+PlGDG8KJDOg2xQJqoqPBzjH/H0MUC03qE1ZPf8uGZa6gL9JsnpRctYLfselhMfsl5b9JxAO3AgZl+S2GAH/mH1BlmwvjjyuGehJmVrVE1r2sviiHCaOf5dZ0h8HCGrco1VqAQKBgQCffYkMofOTSUvjG2mpAHuCOQCsSaDfsFIfSBXQqgUIf7ouc8HAyAM2VOh+NAPj56cRs7opsAxqkvnKc+BoEy8Rdl8RyWeO+qvFNicHelBph9gxeod8SvFIyjsKZ7gwoYf163AIBxMCGeeHLodU5Q10hkv1hau8vv2BcPhdCstu8QKBgQDgO4Rr36Pa5rjbNbGNPsJqALjthTeU0yaU8hdC7Hc7B/T6npEIhw3s1eNq7e0eeDltqz7tDnY8qysazbQZp1cV5TJ8+gtwGmDoADBnU1NXqX4Squfml6OhNEucpTdjux7JdLVmmQFraOT2Eu5f9uuNtA+d8uhBEXhskuvEC552ug==`, 'base64')

subtle.importKey('pkcs8', Buffer.from(pk), { name: 'RSA-PSS', hash: { name: `SHA-384` } }, true, ['sign']).then((pk) => {
  const alg = {
    hash: { name: 'SHA-384' },
    name: 'RSA-PSS',
    saltLength: 384 >> 3,
  }
  subtle.sign(alg, pk, Buffer.from('foo')).then(console.log)
})

[panva]

cc @tniessen @jasnell @nodejs/crypto

[panva]

A usable error message is throw when using crypto.createSign, this is therefore unique to the one shot APIs.

[panva]

I've ported the RSA-PSS tests from test-crypto-key-objects.js to test-crypto-sign-verify.js, see panva@8b7e5c6