Add openssl support as an alternative to mbedtls

[connor-cihula]

Problem description

MbedTLS is prone to introducing breaking changes in minor releases. OpenSSL tends not to have this problem.

Implementation description

OpenSSL can be chosen as the crypto backend with -DCRYPTO_BACKEND=openssl. MbedTLS remains the default, but can be explicitly set with -DCRYPTO_BACKEND=mbedtls.

Screenshots

N/A

Additional things

The OpenSSL implementation refuses to attempt AES modes that MbedTLS does not implement properly.

• XTS: code exists to attempt XTS, but it's wrong and the code isn't even reachable.
• CCM: completely broken, can hang the application.
• GCM: doesn't seem to hang, but incorrect for reasons similar to CCM.
• CBC: padding issues.

[WerWolv]

Thanks for the PR!

However, we used to use OpenSSL in the past before I was made aware that OpenSSL is incompatible with GPL code: #173

Edit: It's possible that that's outdated information, OpenSSL 3 apparently changed the license

[neptuwunium]

openssl changed to apache 2.0 which works with gpl3/gpl2-or-newer but not gpl2.

libressl (bsd's openssl 1.0 fork) also has this issue.

wolfssl is an alternative that does work within the license requirements though.

[neptuwunium]

also, CBC is the most common AES cipher block mode. having the openssl implementation explicitly disable that mode is very problematic.

though a viable alternative is needed as mbedtls, unlike openssl, flirts with the idea to remove ciphers.

Mbed-TLS/mbedtls#9202

[neptuwunium]

does crypto++ satisfy the same scope as mbedtls? that one could work as a suitable alternative.

[WerWolv]

fwiw, I'm also completely fine with staying with mbedtls. I never had any issues with mbedtls upgrades

[billatarm]

fwiw, I'm also completely fine with staying with mbedtls. I never had any issues with mbedtls upgrades

TL;DR this will greatly reduce the amount of effort downstream in Fedora and other distros.

Well, I think for this statement to be true, it depends on how it get's deployed. If you're just building a giant monolith, it's fine. But downstream in packages, they try and decouple the dependencies, and a big one to decouple is the crypto library for a couple of reasons:

1. Update it system wide for all applications
2. System wide crypto policies

mbedtls isn't the best for those scenarios. First, it's designed for monolithic embeded application usage, so the APIs and ABIs change at whim. For instance, in one API stable version, they will re-order structs breaking the ABI. I don't think we remove mbedtls, since it works well for the monolith model, but for Linux distro users, where the "default" crypto provider is OpenSSL, it would be nice to have support here. mbedtls will eventually be removed from Fedora, which means imhex would have to bundle it in the package, which means updates have to be done per-package. In retrospect, and I helped to create this problem, mbedtls should never have made it into the package managers.

[billatarm]

also, CBC is the most common AES cipher block mode. having the openssl implementation explicitly disable that mode is very problematic.

That's why it's in draft right now. So, @connor-cihula's investigation has found that many modes are broken. So we can do one of two things here:

1. remove the broken modes
2. fix the broken modes
3. a combination of both fix and remove

Looking for the community to weigh in, also not sure if anyone is aware of this and fixes are inbound.

[WerWolv]

What's the solution for projects, like ImHex, that can't use the "default" crypto provider though due to its license? I chose mbedTLS in part because of its very permissive license. As it is right now, I can't accept this PR without breaking the GPL

So, @connor-cihula's investigation has found that many modes are broken.

Are they broken in mbedTLS or in ImHex? If it's the latter, I'd absolutely rather fix them

[billatarm]

What's the solution for projects, like ImHex, that can't use the "default" crypto provider though due to its license? I chose mbedTLS in part because of its very permissive license. As it is right now, I can't accept this PR without breaking the GPL

So, @connor-cihula's investigation has found that many modes are broken.

Are they broken in mbedTLS or in ImHex? If it's the latter, I'd absolutely rather fix them

IIUC, within ImHex, do you know of any open bug reports or PRs incoming?

I see these:

• [Bug] AES Decryptor not functioning as expected #1935
• Fix AES ECB mode in Data processor #1986

but I have no idea if it's related.