Fix BIT STRING unused-bits byte in PKCS#8 v2 publicKey (RFC 5958)

[MarkAtwood]

Fixes #10019

The publicKey field of a PKCS#8 v2 private key (RFC 5958 OneAsymmetricKey, [1] IMPLICIT BIT STRING) was written and read without its BIT STRING unused-bits byte. wolfSSL wrote 81 20 <32> for Ed25519 where ring, BoringSSL and OpenSSL write 81 21 00 <32>, and wolfSSL rejected keys from those implementations.

Thanks to @cpsource, whose patch in #10019 (April) took the same approach for the Ed25519, Ed448, X25519 and X448 cases.

Changes

• SetAsymKeyDer writes the 0x00 unused-bits byte before the key, in both the template and original ASN code paths.
• DecodeAsymKey_Assign accepts two forms for fixed-size keys (Ed25519, Ed448, X25519, X448, ML-DSA, Falcon): the RFC form (keySz + 1 bytes, leading 0x00) and the legacy wolfSSL form (keySz bytes). Any other non-empty form returns ASN_PARSE_E; an empty field is treated as absent, as before. Before this, a leading 0x40 or 0x04 reached the Ed25519/Ed448 importers, which accepted it as an OpenPGP or uncompressed prefix.
• CURVE25519_MAX_KEY_TO_DER_SZ and WC_MLDSA_44/65/87_BOTH_KEY_DER_SIZE grow by one byte.

Compatibility

Two effects maintainers should weigh, also stated in ChangeLog.md under Behavioral Changes:

1. Stored keys, forward direction. Keys written by this release cannot be read by earlier wolfSSL. A fleet where the provisioning host upgrades before the devices that read its keys will break. Workaround with no code change: wc_Ed25519PrivateKeyToDer and the other *PrivateKeyToDer functions omit publicKey, and every version reads their output. If maintainers want more, I can add a build macro that restores the legacy write; I did not add one.
2. Shared-library upgrades. The size macros above are compile-time constants. An application built against older headers that sizes its buffer to exactly those values gets BUFFER_E/BAD_FUNC_ARG from *KeyToDer after a drop-in library upgrade. It fails closed, with no overflow, but those applications must be recompiled. The WC_MLDSA_* change may need a decision on the v7 boundary.

X25519 PKCS#8 keys still do not interoperate after this PR: wolfSSL imports and exports X25519 key values in reversed byte order (#11673), so an RFC 8410 key from OpenSSL loads as a different key. This PR fixes only the [1] framing for X25519.

SE050: se050_ed25519_sign_msg imports a software key by passing wc_Ed25519KeyToDer output to the NXP middleware, which already strips a leading 0x00/0x01 from [1]. Under the old layout a private-only key (all-zero public key) failed that parse by accident, which ed25519_asn_test expected as WC_HW_E. With the RFC layout it parsed and signed with a zero public key. The port now rejects a software key without a public key with BAD_FUNC_ARG, the same check as the host path, and the test expects BAD_FUNC_ARG on all builds. Verified with the SE050 simulator (default and WOLFSSL_SE050_ONLY_KEY_ID): wolfCrypt test passes.

Follow-ups

Two pre-existing bugs found during this work have fixes stacked on this PR as drafts: #11675 for #11670 ([0] attributes rejected) and #11676 for #11671 ([1] publicKey not checked against the private key for ML-DSA and Falcon; X25519 waits on #11673). They will be rebased onto master after this merges.

Tests

All vectors come from outside wolfSSL:

• Ed25519: RFC 8032 section 7.1 test 1, RFC 8410 section 10.3, and a ring 0.17.14 generated key and signature. wolfSSL's re-encoding matches ring byte for byte.
• Ed448: RFC 8032 section 7.4 "Blank" key. X25519: RFC 7748 section 6.1 Alice key, checking only the [1] framing (see the X25519 note above). OpenSSL independently derives each public key from its private key.
• Negative cases: unused bits 0x01 and 0x40, a mismatched public key (PUBLIC_KEY_E), a keySz + 2 field, and an empty field.
• The test_SetAsymKeyDer and test_mldsa_der expected sizes are updated for the extra byte.

Built from e61f90d with --enable-opensslall --enable-crl --enable-certgen --enable-ed25519 --enable-curve25519 --enable-ed448 --enable-curve448 --enable-mldsa --enable-falcon --enable-experimental, running tests/unit.test --api:

Config	master	this PR
ASN template	0 failures	0 failures
ASN template, --disable-keygen		0 failures
--enable-asn=original	10 failures	the same 10 failures

The 10 --enable-asn=original failures (ML-DSA PUBKEY/PEM/X509 tests) are already on master. Against the earlier lenient decoder (strip 0x00 when present, pass anything else through), the new test fails: the 0x01 case returns BAD_FUNC_ARG instead of ASN_PARSE_E, and with that case removed the 0x40 key decodes successfully.

[Copilot]

Copilot review overview

🟡 Changes recommended

The newly added X448 decoding branch lacks direct test coverage.

Review effort: Balanced
Findings: 1

Open (1)

• Add test coverage for X448 public-key bitstring stripping · New

What changed in this PR

Corrects RFC 5958 PKCS#8 v2 public-key BIT STRING framing while retaining legacy wolfSSL decoding compatibility.

Changes:

• Adds and validates the unused-bits byte during encoding and decoding.
• Updates affected DER size constants and tests.
• Documents compatibility and migration implications.

File	Description
wolfssl/​wolfcrypt/​wc_mldsa.h	Increases ML-DSA DER size constants.
wolfssl/​wolfcrypt/​curve25519.h	Increases Curve25519 DER buffer size.
wolfcrypt/​src/​asn.c	Implements RFC-compliant encoding and compatible decoding.
tests/​api/​test_mldsa.c	Updates expected ML-DSA sizes.
tests/​api/​test_asn.h	Registers the new ASN test.
tests/​api/​test_asn.c	Adds interoperability and malformed-input tests.
doc/​dox_comments/​header_files/​wc_mldsa.h	Documents ML-DSA behavior.
doc/​dox_comments/​header_files/​falcon.h	Documents Falcon behavior.
doc/​dox_comments/​header_files/​asn_public.h	Documents Curve25519 and EdDSA behavior.
doc/​dox_comments/​header_files-ja/​wc_mldsa.h	Updates Japanese ML-DSA documentation.
doc/​dox_comments/​header_files-ja/​asn_public.h	Updates Japanese Curve25519 documentation.
ChangeLog.md	Records compatibility-breaking behavior.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

MemBrowse Memory Report

gcc-arm-cortex-m4-dtls13

• FLASH: .text +64 B (+0.0%, 192,516 B / 1,048,576 B, total: 18% used)

gcc-arm-cortex-m4-openssl-compat

• FLASH: .text +128 B (+0.0%, 792,412 B / 1,048,576 B, total: 76% used)

gcc-arm-cortex-m4-pq

• FLASH: .text +128 B (+0.0%, 308,720 B / 1,048,576 B, total: 29% used)

gcc-arm-cortex-m4-tls13

• FLASH: .text +64 B (+0.0%, 247,134 B / 262,144 B, total: 94% used)

gcc-arm-cortex-m7-pq

• FLASH: .text +128 B (+0.0%, 309,680 B / 1,048,576 B, total: 30% used)

gcc-arm-cortex-m7-tls13

• FLASH: .text +64 B (+0.0%, 247,134 B / 262,144 B, total: 94% used)

linuxkm-pie

• Data: __patchable_function_entries +16 B (+0.1%, 28,680 B)
  No memory changes detected for:
• gcc-arm-cortex-m0plus
• gcc-arm-cortex-m3
• gcc-arm-cortex-m4
• gcc-arm-cortex-m4-baremetal
• gcc-arm-cortex-m4-crypto-only
• gcc-arm-cortex-m4-min-ecc
• gcc-arm-cortex-m4-pkcs7
• gcc-arm-cortex-m4-rsa-only
• gcc-arm-cortex-m4-sp-math
• gcc-arm-cortex-m4-tls12
• gcc-arm-cortex-m7
• linuxkm-standard
• stm32-sim-stm32h753