Repository navigation
Botan4 #4666
Description
Botan4 Tracking Issue
It took almost 2.5 years from when development stopped on Botan2 until Botan 3.0 was released. That was suboptimal.
Goal this time around is that the entire Botan3->Botan4 development phase happens in a single release cycle. In one quarter there is a Botan 3.y.0 feature release, same as every quarter. The next quarter sees the release of Botan 4.0.0, plus a Botan3 patch release as required.
This requires effectively pre-loading almost all of the development work in advance, such that once master moves to 4.0-pre the process is mostly just hitting merge-merge-merge on a pile of already existing PRs. Thus the very long lead time (and the desire for a memorable ticket number, since this meta issue will be open for years - still sad I missed out on #4444).
Note
For issues or questions related to specific subtasks, please open up a new issue for discussion, and reference this ticket. Otherwise, this issue is likely to become clogged with many unrelated conversations.
General Outline
Botan4 is still C++20. The gains we'd get from C++23/C++26 seem minimal (not zero, but nothing amazing) and language bumps do have an effect on end users. Hopefully MISRA eventually allows C++20...
Increasing compiler versions TBD. Certainly Clang minimum version increases, so we can use std::source_location (prior work done by @KaganCanSit that could be resurrected: #5084). Increasing minimum GCC to 14 would ensure we always have __builtin_addc which would be useful for optimization purposes. At least Clang 17 for the CWG 2518 fix.
Timeline
TBD. Maybe mid 2027?
Significant Development Work
The bigger projects
- Split public key and private key types
- Remove DHE support from TLS
- Remove RSA key exchange from TLS
- Remove CBC ciphers from TLS
- Removing support for renegotiation in TLS 1.2/DTLS 1.2
- Post removal of kyber_90s and dilithium_aes some cleanup are possible
- Possibly ML-KEM vs Kyber and ML-DSA vs Dilithium cleanups
- Support large element OIDs
- Internal EC data cleanups after removing BigInt EC point logic
- BigInt public API cleanups
- Resurrect changes in Replace BOTAN_FFI_VISIT macro with std::source_location (use botan_ffi_visit function) #5084
For some of these it may be possible to do significant amounts of work on master beforehand to get ready, which will make life simpler. In particular for DHE/RSA/CBC in TLS it may be possible to just make them optional (ie tls does not hard depend on dh module); this improves things on master immediately for those who would prefer a smaller attack surface, and makes the final incompatible change easier. Splitting the key types and the ML-KEM/ML-DSA cleanups likewise I think a lot of prep can be done without violating SemVer.
Incompatible Changes
Misc mostly easy changes, no need to preload these can probably all be done over a weekend.
- Removing ~all deprecated functions/classes
- Remove the deprecated elliptic curve groups
- Remove the deprecated DL groups
- Decide how much of explicit curve support to drop #4684
- Remove serialization/deserialization of EC identity element
- Remove PBKDF and subclasses
- Hide PasswordHash headers
- Remove deprecated PK padding aliases
- Headers going internal: numthry.h, reducer.h, compiler.h
- Underscore prefix all internal functions (eg
create_encryption_op) [also consider using some nasty_botan_internal_prefix instead of just_] - Rename ML-DSA parameters to match FIPS 204 (ML_DSA_4x4 -> ML_DSA_44) (Add NIST names as further alias for ML-DSA #5270)
Module Removal
If you, dear reader, are relying on any of these modules/algorithms in your code, please asap open a sub-issue for discussion, so we are aware of your usage. Do not reply to this ticket about specific modules.
- Decide if
cryptoboxshould be removed in Botan4 - Decide if
dilithium_aesshould be removed in Botan4 - Decide if
dliesshould be removed in Botan4 - Decide if
gost_28147should be removed in Botan4 - Decide if
gost_3411should be removed in Botan4 - Should
keccaksupport be removed in Botan4 #4721 - Decide if
kyber_90sshould be removed in Botan4 - Decide if
legacy_ec_pointshould be removed in Botan4 - Decide if
lionshould be removed in Botan4 - Decide if
mceshould be removed in Botan4 - Decide if
md4should be removed in Botan4 - Decide if
noekeonshould be removed in Botan4 - Decide if
shake_ciphershould be removed in Botan4 - Decide if
tpm(and alsouuid) should be removed in Botan4- Note
tpmis the TPMv1 only module,tpm2supporting TPMv2 is staying
- Note
Currently deprecated but not removed
These are already deprecated and certainly not desirable but kept for Botan4
crc32dsagost_3410md5siphashstreebogx919_mac
Activity
crazydef commentedon Feb 13, 2025
Completely removing modules will make people reconsider Botan as a reliable, stable library moving forward. Myself included.
While I appreciate algorithms become less secure over time, many applications need to maintain compatibility with other systems that still use them - for whatever reason. Being able to validate a CRC32 or MD5 hash, for example, that has been received from another application/system is still a valid use case. (Or even using insecure algorithms in systems that do not require absolute security - such as file hashing for data storage in games, for example.)
May I humbly recommend that they're disabled from the builds by default, but can still be re-enabled for those who really need them?
randombit commentedon Feb 13, 2025
By clearly documenting "X is deprecated and may be removed in a future major release" and then, years later, removing X in a future major release?
You make an argument for CRC32 and MD5. Which are pretty plausible I guess. Can you make a similar statement about Lion, Noekeon, or HyMES McEliece? How about other algorithms which similarly have been added and later removed like Blue Midnight Wish, DESX, Square, Skipjack, FORK-256, CS-Cipher, SHARK, ThreeWay, Rabin-Williams, EMAC, MISTY1, Kasumi, or CAST-256?
Already the cost of removing a feature is high; once something is added we have to maintain it for years and years, at least until a new major release. If the cost of removing a feature is literally infinite - once added we are stuck with it forever - then the bar to adding an algorithm becomes equally high. For example the preliminary versions of Kyber and Dilithium would not have been merged, if it was impossible to remove them later. Instead we would have waited (years!) for the final NIST competition results. And only then would we have been able to start work on post quantum secure TLS.
If there is a specific algorithm (say CRC32 or MD5) that you wish to argue against removal please do open a subticket and we can discuss. I am very open to arguments of the form "I am currently using, or might in the future want to use, (this specific thing) and removing it will be an inconvenience for me." (see for example #4678 or #4685) I am however not interested in an argument of the form "You cannot remove anything ever for any reason." If you have been operating under the impression that - in a major release - we would never remove some number of {experimental,insecure,obsolete,obscure} algorithms, after a clearly documented deprecation phase, then I regret to inform you that this may not be the library for you.
That could be a path to take for something like MD5, which is small and self-contained. Many of these, their existence implies significant ongoing maintenance burden because, compiled by default or not, they need to compile. For example supporting 90s mode in Kyber requires contorting the whole implementation.
crazydef commentedon Feb 13, 2025
Obviously there's a trade off between maintainability and backwards compatibility. And I'm not suggesting that everything should be kept unnecessarily.
But that said, if something is self contained and can be easily kept, personally speaking, I would try to keep it. I might even be tempted to break out chunks of code that isn't easily separated and keep them in a v3 compatibility module.
My original point being people who rely on third party libraries should be able to rely on them. There's nothing worse than having to waste time rewriting huge chunks of code just because someone else (on the team, not taking a swipe at you here) decide they will only ever work with the latest and greatest of every third party library in the system. I work with a few people like that already, and it's not fun constantly playing catch up.
But this is not my project, and you are free to do what you like, obviously. As someone who has recently switched to Botan from another library, I just wanted to make my thoughts known. :)
randombit commentedon Feb 14, 2025
I am not trying to make peoples lives hard here. Literally a single random person showing up anytime in the next two years and saying "I am actively using X" is sufficient for something to be struck from the list of potential changes. My actual belief is that (outside of possibly DSA, MD5, X9.19 MAC and CRC32) there are literally zero users of these anywhere.
Takarth commentedon Feb 27, 2025
keccaksupport be removed in Botan4 #4721Chaosvex commentedon Mar 22, 2025
Another vote for keeping MD5 and CRC32 (specifically). As mentioned, there are still plenty of protocols (e.g. STUN) that use them and it'd be unfortunate to have to drag separate implementations in, even if it wouldn't be the end of the world. I'm assuming deprecated* features aren't going to trigger compile warnings, because that'd also be unfortunate.
*although I'd argue against deprecating them at all.
oviano commentedon May 13, 2025
I'm c urious as to where some work on DTLS fits into this, e.g. DTLS 1.3, or at least fix some of the bugs in DTLS 1.2 (such as #4782)?
ethindp commentedon May 30, 2025
My thoughts as someone who loves botan:
As for the other removals, I don't use much of them so don't really have a steak. But I'd love to see DTLS improved, if that's possible (I know it has some flaws which may be due to the protocol itself and not any crypto impl but I could be wrong).
Would it be worth adding the Noise framework? The only one I know of that's in modern C++ is mine, but it's far from perfect and could use a lot of improvement. But if that's too far out of scope that's fine too.
randombit commentedon May 30, 2025
It's an unfortunate one because it is still, sadly, in use in various niches. This is a common element to the remaining we-really-should-remove-this algorithms -- the obscure and rarely implemented ones were already removed years ago.
The TPM deprecation is for TPMv1, not affecting TPMv2. TPMv1 is restricted to SHA-1 only, and isn't included on modern systems.
Yes Noise framework is absolutely in scope, but there is no need there to wait for Botan4, new features are added all of the time. The major version bump is only because we're removing features and/or breaking APIs.
DTLS absolutely needs some love and I have not had the time to dedicate to it recently.
ethindp commentedon May 30, 2025
Yeah, completely understandable. I'd love to see it gone, but there are still (and probably always will be) some valid use-cases for it, even if niche.
Aha. That should probably be clarified in the original comment -- it reads (at least to me) as though all TPM support is possibly going to go.
Ah okay. I'd be happy to help with Noise, or you can just use the impl I wrote (it's public domain and not too hard to integrate/use). Shameless plug, I know.
It's hard for me to actually come to a conclusion on DTLS, particularly given there seems to be a lot of conflicting information about it. Some sources say it's bad and shouldn't be used, others say it's fine... Yotta yotta yotta. Anyway, would be great to see it get some love at some point.
oviano commentedon May 30, 2025
DTLS in Botan works pretty well for me. There are a couple of documented issues where it breaks on a flakey connection due to the transmission logic which necessitates closing and re-opening the connection to try again and it would be great if it was upgraded to support DTLS 1.3 but it's perfectly usable as is.
3 remaining items
reneme commentedon Feb 13, 2026
Now, I might be naive, but I really hope that C++20 isn't blocking anyone in 2027 anymore. Rolling back would certainly be painful.
ethindp commentedon Feb 13, 2026
So maybe this could be something added in Botan 3. but JWT support would be nice. Maybe OAuth 2.1 would be a bit much. Like, no, not asking for a full HTTP server. Just the plumming. Like we do with the TLS stack: the app implements the actual IO logic, and Botan handles the hard/complicated annoying things. But this might be too much.
falko-strenzke commentedon Feb 19, 2026
@randombit
@reneme
From the lists of things to remove it seems DilithiumRound3 and KyberRound3 are not part of it. If those are going to stay, I would suggest some changes, for instance type separation between the NIST and the round 3 variants (which are currently the same key type with alias definitions). We could do that as part of the current BSI project. If the round 3 variants are going to be removed anyway, it wouldn't make much sense to invest time and effort here.
randombit commentedon Aug 15, 2026
@falko-strenzke Sorry just saw this note. I think at the time of this writeup the R3 variants were not deprecated (only the 90s versions), but they are now, and I believe Botan4 should be final ML-KEM + ML-DSA only, without any support for the R3 variants.
reneme commentedon Sep 15, 2026
Suggestion: Separate Key Encoding from the Asymmetric Key API
I think @falko-strenzke has a point with this comment (and there's more related discussion in this pull request). TL;DR: Let's separate asymmetric key encoding from the actual asymmetric key objects entirely. Much like
PK_Signer,PK_Verifier, and friends separate key operations into generic helpers. I think we should introduce a builder-configurablePK_Encoder(and maybePK_Decoder, I'm not sure yet) to replace the ad-hoc per-algorithm methods we maintain in specific algorithms.Inconsistent public/private key encoding formats for different algorithms keep being a challenge. Some examples:
@randombit If there's no general objection to this approach, I would open a separate issue to discuss details. (Or did I perhaps even miss an existing issue that tackles this already?)
randombit commentedon Sep 15, 2026
Yeah makes sense. This is partially addressed in #4593 but only for the PKCS8 parts. I don't have a specific vision on this (at least not yet, I'm sure I'll have Opinions on the PR) but one thing I would love to be able to remove in Botan4 is
EC_PublicKey::set_point_encoding[along with the already deprecatedset_parameter_encoding] - a key having mutable state of how it is encoded is pretty gross imo.Interestingly this might even interplay with the request further up thread for JWT support...
randombit commentedon Sep 15, 2026
Re the encoding options we could likely introduce it already in Botan3, which would be nice if possible in that it provides a reasonable transition path (making it is possible to use the same code against recent-enough-Botan3 as well as Botan4, rather than introducing a completely clean break where you have to have two distinct codepaths based on the major version)
ethindp commentedon Sep 15, 2026
I have another request: would it be possible to allow decoding of PFXs (e.g., that generated mainly by Windows)? Right now I have to combine Botan and OpenSSL and that leaves a bitter taste in my mouth. I guess I could just hand-role it but that doesn't seem like the smartest idea.
As for Noise support, I wouldn't mind contributing it, although I would probably want it to be thoroughly audited. I've written a noise implementation before (https://github.com/ethindp/noise-cpp) and Botan would significantly expand the list of ciphers and hash algorithms. Could probably take significantly from the Noise implementation I wrote, although it certainly wouldn't look even remotely the same.
randombit commentedon Sep 15, 2026
@ethindp PFX/PKCS12 support is available starting in 3.13 so you should be all set, with the perhaps notable exception that right now we don't support RC2-encrypted PFX
mouse07410 commentedon Sep 16, 2026
falko-strenzke commentedon Sep 21, 2026
mouse07410 commentedon Sep 22, 2026
mouse07410 commentedon Sep 23, 2026
randombit commentedon Sep 23, 2026