OpenSSL legacy provider isn't always installed on Debian unstable #11450
Comments
|
I guess as a starting point: What would your desired behavior be :-) I'm guessing it's something like: if My suggestion would probably be to just change cryptography to silently allow the legacy provider to fail to load. This would look something like (untested): diff --git a/src/rust/src/lib.rs b/src/rust/src/lib.rs
index cd7b99f15..dce24b100 100644
--- a/src/rust/src/lib.rs
+++ b/src/rust/src/lib.rs
@@ -60,9 +60,7 @@ fn _initialize_providers() -> CryptographyResult<LoadedProviders> {
.map(|v| v.is_empty() || v == "0")
.unwrap_or(true);
let legacy = if load_legacy {
- let legacy_result = provider::Provider::load(None, "legacy");
- _legacy_provider_error(legacy_result.is_ok())?;
- Some(legacy_result?)
+ provider::Provider::load(None, "legacy").ok()
} else {
None
}; |
|
It occurs to me that this will also impact our non-wheel users. If distros are going to be switching legacy to not be installed by default, I guess we'll probably also need to handle legacy failing to load silently. |
|
Yes, graceful failure sounds like the best possible option here. |
|
I'm not completely sure what my desired behaviour would be! It does include not having cryptography be weirdly different on Debian than elsewhere, though I appreciate that's a bit vague. Silently allowing the legacy provider to fail to load is probably OK, although it does mean that users of cryptography-based applications who don't have |
|
I assume that the overwealming majority of cryptography's use will not need the legacy provider, and wouldn't want to be showing warnings. It's going to be things like exhaustive test suites that need it. |
|
I would not assume that. AFAIK legacy provider is needed for several algorithms that are widely used in PEM and PKCS#12 encryption. (These algorithms are garbage are people shouldn't use them, but here we are.) FWIW the original motivation for making it noisy for the legacy provider to fail to load was concern that users would basically bork their installs and it'd be impossible for them to debug. Do you know if any other distros are planning to put the legacy provider in a separate package? |
I'm afraid I have no idea. If any of the Debian OpenSSL folks show up here then maybe they will. |
|
Without the legacy provider RC2 PKCS12 encryption (which is, sadly, incredibly common) will fail with an opaque error: cryptography/src/rust/src/pkcs12.rs Lines 719 to 721 in 45f0c8d As Alex noted there are also other scenarios where this is likely to happen. I'm generally supportive of silently not loading the legacy provider, but I think we need to make the error messages where these failures can occur actionable. This probably requires error stack parsing though, and error stacks aren't guaranteed stable (nor are they always consistent across OpenSSL, LibreSSL, and BoringSSL, sigh). I suppose one other approach would be to simply track whether we've loaded legacy or not and just add some alternate text to the error, e.g. " or possibly due to lack of legacy provider" |
|
Also, I wonder if "widely used in PEM and PKCS#12" may be a reason why Debian's OpenSSL team should reconsider their packaging structure? The original motivation in @xnox's bug report said "None of the algorithms it provides are useful, or needed at all", which is strongly at odds with what you're saying here. Perhaps they were unaware of how widespread the uses actually are? |
|
I have been hit by this in Wolfi OS / Chainguard too. IMHO the variable as is doesn't make sense (opt into secure behaviour). Standards compliance and certification require to block this. Thus I am literally shipping docker containers that set said variable, because with strict FIPS mode enabled it just crashes. Even if legacy provider is present. I am trying to go even further than just splitting. IMHO all distros should start shipping opensslconf.h setting / indicating that none of these things are available anymore. Such that without breaking runtime ABI, we can make all runtime applications to stop using these things. RC2 PKCS12 is museum cryptography. Upstream calls it pathetically weak. Please convert your private keys to anything stronger or unencrypted and protected by other means. |
|
@xnox You should re-consider your approach, both maintainers of this library had extremely negative reactions to your comment. And the reason for that is that you've failed to understand how ecosystems works, and instead loudly demanded things from people who cannot give you them. We support RC2 in order to parse things emitted by other projects. We don't ourselves encrypt anything with RC2, and haven't for a long time (if ever). We're not at all confused about how cryptographically garbage it is. However, RC2 remains widely used. The macOS still uses RC2-40 for encrypting the certificate container in PKCS#12. It's also what OpenSSL<3.0 used, most examples for libraries like bouncycastle continue to show, Windows uses it in some circumstances, etc. Looking around at the current state of this ecosystem you'll find numerous examples using the Most people don't use PKCS#12 encryption as a security boundary, they use it as an interchange format. So we're as interested in anyone else in getting rid of RC2, however we also need to manage our breakage budget (see https://alexgaynor.net/2021/oct/07/whats-in-a-version-number/), and breaking common workflows like "parse a PKCS#12 from macOS" for limited security gain. If you want to get rid of PKCS#12, the place to start is with people who emit it, not people who parse it. |
|
Supporting converting RC2-40-CBC files to AES-256-CBC makes sense on Linux as a standalone tool. Requiring runtime cryptographic libraries to support direct usage of RC2-40-CBC certificates without requiring to upgrade them first does not make sense. The risk of having legacy provider available and loaded is too large causing to unintentionally use weak cryptography. Ditto all secret scanning and security scanners should flag up weak private keys. The common workflows must change to add a new step of converting private keys to strong encryption prior to usage. We must protect user privacy, even if Mac OS chooses not to. I will also email Apple to address their defaults. |
|
I would like to come to a situation where the legacy provider is only loaded in cases where the maintainer knows about it, and that it's needed, for instance the application that needs to do PKCS#12 with RC2. I do not want something RC2 to be available in all applications. So loading it when it's available is also not a preferred solution for me, I would like to push it application. |
|
As a general purpose cryptography library, we don't really know what our users are going to use us for, which makes it more challenging to only load it when required. (If Python packaging had some sort of "features" indicator, ala Rust's Cargo, we could use it to opt-in, but it doesn't.) |
Assume that OpenSSL on some distributions no longer provides RC2. Meaning try not to go via OpenSSL to gain RC2. Most of the time it should not be needed. And when that fails, fallback to a vendored in implementation of RC2 to on-the-fly convert to unencrypted or AES-256-CBC to then continue back via OpenSSL or anything else. This is similar to how other pieces of software have adapted to continue using obsolete cryptography, as required to maintain interoperability with whatever you want to support. I.e. I have seen projects vendor in MD4, MD5, RC2, RC4 to continue supporting NTLM auth and similar things. Crashing upon importing your library, because legacy provider is not available, is not optimal for all the users who are on modern linux distributions, and are not planning on using weak cryptography (which obviously is impossible to know ahead of time). As the current variable is "opt out of legacy algorithms support", instead of "opt into legacy algorithms support", and thus forcing systems to carry additional runtime dependencies which depending on use case will never be used (with a declining probability over time). |
|
Vendoring legacy cryptography functions in 2nd-level libraries to allow core cryptographic libraries to drop support for them, seems.... not particularly useful |
|
It also simply wouldn't work. RC2 decryption is handled by OpenSSL as part of the PKCS#12 parse-and-decrypt process, there's not an entrypoint for us to only do that RC2 encryption. |
|
Debian has reverted this change for now, to figure out next steps without breaking everything. |
|
Would it be possible to only load the legacy provider if some function is called? |
|
About Mac OS X: Version history is available at https://en.wikipedia.org/wiki/MacOS_version_history Note that Catalina release went EOL in 2022, but macOS releases can be very sticky even when hardware supports newer releases, as one may simply not have enough disk space to upgrade, nor a compelling reason to upgrade. Supported releases of Mac OS do support AES-256-CBC. Obviously keys generated in previous releases, are not converted automatically. However, this incompatibility with Mac OS X should be diminishing over time, as people rotate keys / generate new keys on the supported Mac OS X releases. |
|
I just exported a cert+key from Keychain Access on macOS 14.6.1 (latest release) and it used RC2-40 and 3DES. I also tested on a machine running the beta release of Sequoia that was released today -- RC2-40 and 3DES. So while they may support newer algorithms on import, they are still choosing to export legacy cryptography (with no UI options to change it). |
Absolutely fabulous. Probably likely due to appleTV and iPhone support - as I think those devices have longer support time-frames and thus more of them are behind the curve, requiring RC2-40. How broken is RC-40 and 3DES and how quickly those keys can be bruteforce decrypted, to effectively render them plain text? Sort of pondering if a javascript web-app can do it. |
|
@reaperhulk if you can make a gist of a freshly generated & exported key, that would help with cracking it or providing instructions on how to convert it. As I think at the very least OpenSSL documentation should direct people at upgrading those. @reaperhulk are you able to test if import of strong pkcs12 file works via gui in the keychain app? (one can generate one with mac os terminal). |
|
Let's try to keep this thread on topic. For the purposes of this thread, PKCS#12 + RC2 is a fact of life, we can focus on changing life elsehwere. @kroeckx to answer your question, I think theoretically we could lazily load it, but it'd be anytime PKCS#12 was used, not just when RC2 was used (and anything else that might require these algorithms, which is maybe all PEM loading?). We'd still have to decide what the behavior is (allow it to silently fail or not). And finally, I think the fact that OpenSSL allows lazy-loading of algorithms is a very poor design decision on their part, and the shared mutable state they have has led to tons of performance regressions, so I'm very reticent to rely on it. |
alex
added
the
waiting-on-reporter
|
This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days. |
|
Would it be useful to have a compile time flag that explicitly disabled the legacy provider? I work in an context where we don't have the legacy provider within our OpenSSL conda package, and currently use a tiny patch to disable Thanks for all the work that goes into |
|
What are you imagining in terms of how the user would configure that legacy
is disabled?
…On Mon, Sep 23, 2024 at 11:51 PM Shaun Walbridge ***@***.***> wrote:
Would it be useful to have a compile time flag that explicitly disabled
the legacy provider? I work in an context where we don't have the legacy
provider within our OpenSSL conda package, and currently use a tiny patch
to disable legacy explicitly. For those who are in environments where
they explicitly want legacy primitives to fail and are in a position to
build from source, this would provide a mechanism to enforce that desire.
Thanks for all the work that goes into cryptography!
—
Reply to this email directly, view it on GitHub
<#11450 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBG4ECVO2WLVS6VLUUDZYDOUNAVCNFSM6AAAAABMWTVRIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZQGA3DMNRXGE>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
I admit I have limited knowledge of the modern pip install workflows, but possibly by setting an environment variable at build time, e.g. reuse the same environment variable with: export CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1
python -m pip install . -vvSome existing downstream builds already set |
|
We could do an env var. My instinct is that it should have a separate name.
I'm happy to take a PR for this
…On Tue, Sep 24, 2024, 10:34 AM Shaun Walbridge ***@***.***> wrote:
I admit I have limited knowledge of the modern pip install workflows, but
possibly by setting an environment variable at build time, e.g. reuse the
same environment variable with:
export CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1
python -m pip install . -vv
Some existing downstream builds already set OPENSSL_DIR in a similar
fashion when building cryptography, which is bound to within the
openssl-sys package
<https://github.com/sfackler/rust-openssl/blob/openssl-sys-v0.9.72/openssl/src/lib.rs#L55-L56>
.
—
Reply to this email directly, view it on GitHub
<#11450 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBCLU42EBIAH7K5IB7DZYFZ53AVCNFSM6AAAAABMWTVRIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNZRGQ4DEMBRGY>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
|
This issue has been waiting for a reporter response for 3 days. It will be auto-closed if no activity occurs in the next 5 days. |
|
This issue is labelled as waiting for a response from the reporter (me), but I'm not really sure what I can provide given that I was just trying to link up two other parties (cryptography upstream and the Debian OpenSSL team). |
alex
removed
waiting-on-reporter
|
I think we've accomplished that :-) I'll hold it open to track adding a build-time env var. |
|
build-time env var? How is it going to work? Sebastian |
|
Users who opt into building without legacy support will be unable to ue
legacy algorithms.
…On Sat, Oct 5, 2024 at 6:32 AM sebastianas ***@***.***> wrote:
build-time env var? How is it going to work?
I though something in the cryptography function that will ask for the
legacy algorithm and if it gets none, it is just quiet and complains once
the algorithm is actually used. Or is this not feasible?
If you remove support for legacy algorithms (which is my maybe wrong
interpretation) then users won't be able to open certain certificates.
Sebastian
—
Reply to this email directly, view it on GitHub
<#11450 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBGQ2EEK334ZPUPVR6TZZ7E5XAVCNFSM6AAAAABMWTVRIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGOJVGAZDMNRTGU>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
On 2024-10-05 06:05:14 [-0700], Alex Gaynor wrote:
Users who opt into building without legacy support will be unable to ue
legacy algorithms.
I am not sure if this helps.
Debian will likely build it once with legacy support and depend on the
legacy provider library. Just to simplify things for the user. That
means the legacy provider remains.
Providing two packages (with and without legacy support) will likely
complicate things and people won't know which one to use.
This runtime "complaint" once the algorithm is asked and not available,
is not possible? Excuse me asking this again but I have't look into the
source nor would I describe myself as fluent in Python.
Sebastian
|
|
The desire for building without legacy support is, as I understand it, orthagonal to the request from Debian. We have a few choices available to us:
I'm somewhat persuaded that the right direction is: eagerly load "legacy" and emit a warning if it fails, provide a run-time option to disable attempting to load legacy, as well as a compile-time option to disable loading it ever. But I'd like to hear if a) @reaperhulk agrees, b) that sounds like it addresses various folks requests, and if so if anyone would be interested in submitting a pull-request for it. |
|
There is case 5 - on operating systems in FIPS approved mode obtaining legacy will be blocked. However, legacy algorithms maybe still available at runtime via fips module for approved legacy operations - for example decryption of a private key. |
|
Anything beyond "if/how/when is legacy loaded" is beyond the scope of the
issue. (OpenSSL's design here is _bad_ and dealing with this issue makes me
look forward to a day when using some other library is practical.)
…On Wed, Oct 9, 2024 at 4:17 PM Dimitri John Ledkov ***@***.***> wrote:
There is case 5 - on operating systems in FIPS approved mode obtaining
legacy will be blocked. However, legacy algorithms maybe still available at
runtime via fips module for approved legacy operations - for example
decryption of a private key.
—
Reply to this email directly, view it on GitHub
<#11450 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBCP6Q7C4OVXF6HRTJDZ2WFMPAVCNFSM6AAAAABMWTVRIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBTGM2TEMRTHA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
I am okay with an eagerly load but emit warning if unavailable approach, although ideally we'd also augment our exceptions to include a bit more info in their repr so we can more easily detect failures due to a missing legacy provider. |
|
Sounds good. Does that solve other folks' issues? Is anyone interested
in submitting a PR for this behavior?
…On Wed, Oct 9, 2024 at 4:36 PM Paul Kehrer ***@***.***> wrote:
I am okay with an eagerly load but emit warning if unavailable approach, although ideally we'd also augment our exceptions to include a bit more info in their repr so we can more easily detect failures due to a missing legacy provider.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
|
FreeBSD also builds OpenSSL 3 without Legacy Provider. I find it kinda weird that a program does not work, because something it does not need is not available. Emitting a warning when Legacy Provider aren't available is ok. (In my opinion this would be still the wrong behavior, but i understand why it would be necessary) |
|
Would you be interested in contributing a PR implementing the behavior
described above?
…On Wed, Nov 20, 2024 at 7:43 AM MGlaus ***@***.***> wrote:
FreeBSD also builds OpenSSL 3 without Legacy Provider.
I find it kinda weird that a program does not work, because something it
does not need is not available.
Emitting a warning when Legacy Provider aren't available is ok. (In my
opinion this would be still the wrong behavior, but i understand why it
would be necessary)
—
Reply to this email directly, view it on GitHub
<#11450 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBF66RSRS3VDI2PO3P32BR7YVAVCNFSM6AAAAABMWTVRIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIOBYGQ4DQNBWGA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
Sorry, i don't have the necessary experience for it. I don't know any rust and i don't have much experience in python too. |
|
Is there more to this than just legacy.
Above will get more weird at runtime over time. I.e. Soon, SHA1 will be available, blocked to generate a digest, allowed to generate HMAC with long enough key; blocked to generate a signature; allowed to verify a signature. Emitting a warning at that point may be helpful; or possibly raising a error. Cause human guidance should state, what happened, why, and what actions a user can take if any (i.e. install additional providers, use a different build of openssl, update cryptography). Existing openssl errors really should be bubbled up verbose (key size too short; digest not allowed; digest not know; curve not known; etc). Such that users of pycryptography APIs can handle them intelligently (fix their deployment, downgrade/rebuild openssl, or in their app tell their user what happened and what they should do). I haven't looked, but this may require adding more error handling in the openssl bindings; and then also adding those warnings/errors in python cryptography; and emitting lots of warnings/errors (and allow to catch them, or suppress them). Because yes most of the time, most things, just work. But there are many many many combinations that do not and will not (i.e. when there is no legacy provider at all, or openssl was compiled with no-ec to completely remove ECC support, and so on). |
|
You're conflating unrelated issues. The precise behavior of different APIs when algorithms are unavailable (either entirely or for specific operations) are specific to those APIs, but in general we almost certainly already raise a clear exception for all of those, and where we don't there are individual bugs to be filed. This bug is specifically about our behavior of automatically attempting to load the legacy provider and erroring if it is unable. I will also take this moment to register my firm belief that the providers API was a terrible design on OpenSSL's part, and some day in the future when I finally snap and propose we drop OpenSSL, providers will have played a substantial role. |
Previously it was stated that hard loading legacy provider is done to ensure eventual availability of algorithms. Loading legacy provider doesn't change any available C APIs - e.g. with or without legacy provider EVP_Digest APIs are available. Reasons for force loading it are unclear to me, as it will be autoloaded by openssl when needed regardless. E.g. If one makes request for EVP_Digest of NID MD4, and no imolementstion is found, legacy provider will be attempted to be autoloaded, and if not available error out. Furthermore, depending on default property query string, even with a loaded legacy provider, particular algorithm access may still be blocked. I struggle to understand reasons for manually loading legacy provider at startup 1) when it will not be needed at all 2) may not actually do anything 3) as intended by system administrator / OS. Why is legacy provider being loaded manually at all? Is it because runtime errors were not clear to the user and pycryptography was getting blamed? Is autoloading legacy peovider in openssl broken on some distributions? |
|
I am reading cipher_registry.rs and the implementation there assumes that build-time openssl configuration is the same as runtime. However, that is not universally true. Instead of hard coding the list of things based on openssl build time config, one really should at init time query what algorithms are available and register those dynamically and expose them to the Python world. I.e. there shouldn't be any need for A good example is supporting Blake3 & GOST algorithms dynamically at runtime, even though neither are available in stock openssl nor stock providers. If one queries available algorithms at runtime, it should just work. |
|
Any updates on this issue? |
|
This issue is 100% up to date with both the agreed upon future direction
and the absence of any volunteers to implement it.
…On Wed, Dec 4, 2024, 8:42 AM MGlaus ***@***.***> wrote:
Any updates on this issue?
—
Reply to this email directly, view it on GitHub
<#11450 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBF5G6OSZYTPRPTQUPD2D4BDRAVCNFSM6AAAAABMWTVRIGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJXGQZTMMZYG4>
.
You are receiving this because you commented.Message ID:
***@***.***>
|
This is an attempt to get a conversation started. I wasn't responsible for the original change here, but I've been cleaning up after it in Debian's Python team and I'm not convinced my current approach is really ideal.
The OpenSSL packages in Debian unstable recently changed to move the legacy provider into a separate package,
openssl-provider-legacy. There's aRecommendsrelationship from the main library package (libssl3t64) which means that it's installed on typical user systems, but it's not aDependsso it's now possible for users not to have the legacy provider installed. In particular,Recommendsare not normally installed in package builds and automatic test environments (deliberately, to ensure that declared dependencies are sufficient), and as a result we find ourselves having to apply workarounds to a fairly large number of Python packages in Debian to get their tests working again.I'd like to find out what approach you'd recommend here. I can think of a bunch of possibilities:
CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1in the Debian build rules for lots of Python packages (this is pretty much what I'm doing at the moment, but it's repetitive)openssl-provider-legacy(also repetitive and feels heavy-handed)python3-cryptographypackage depend onopenssl-provider-legacyRecommendstoDepends)cryptography/src/rust/src/lib.rsin some wayDo you have any opinions on this? Ideally I wouldn't be passing messages back and forward, so I'll see if I can get Debian's OpenSSL maintainers to show up on this issue.
The text was updated successfully, but these errors were encountered: