SystemError in OpenSSL's SHAKE when passing negative digest sizes

[picnixz]

Bug report

Bug description:

import _hashlib

_hashlib.openssl_shake_128(b'').digest(-1)     # SystemError
_hashlib.openssl_shake_128(b'').hexdigest(-1)  # MemoryError

import _sha3
_sha3.shake_128(b'').digest(-1)                # ValueError
_sha3.shake_128(b'').hexdigest(-1)             # ValueError

The reason is that OpenSSL implementation accepts a ssize_t but HACL* accepts a uint32_t. I suggest raising a ValueError in OpenSSL's implementation as well. I'll rewrite #135744 for this issue at the same time.

Now, to prevent users passing incorrect data length, I also suggest to restrict has lengths to 2 ** 29 as it was done in HACL* and in #79103 but I'll do it in a follow-up. In the first time, let's only focus on raising a ValueError for all implementations on negative inputs.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-135759: consistently reject negative sizes in SHAKE digests #135767

Note

See #135759 (comment) for the rationale of a non-backport.

[picnixz]

I'm not going to backport this as this changes error messages and exception types. AFAIK, there is no crash, just weird exception types.