Dependency on rust removes support for a number of platforms

[mgorny]

I would like to report that the newly added dependency on Rust has made it impossible to package cryptography for a number of supported Gentoo architectures (and these are architectures where people actually use Python packages that depend on cryptography).

Please see Platform Support. Besides degrading a few of our targets to 'Tier 3', Rust completely does not support at least alpha, hppa, ia64, m68k, s390 (not -x). The cost of adjusting cryptography's C code to these platforms (even if we assumed they wouldn't work out of the box as they do so far) is much less than the cost of porting the whole Rust. You can't really expect volunteers to port the whole Rust to other architectures just to get cryptography back.

Right now it's a really hard struggle to even get Rust packaged for non-Tier 1 architectures (i.e. where upstream does not provide bootstrap binaries).

[LindezaGrey]

destroyed my CI/CD Pipeline this morning 😆

[clanzett]

It is also impossible to build on ANY alpine linux version/architecture. Please remove this dependency as soon as possible.

[tiran]

It is also impossible to build on ANY alpine linux version/architecture.

Update to most recent Alpine with Rust >= 1.45. Alpine latest has rust (1.47.0-r2).

Please remove this dependency as soon as possible.

Not going to happen. Rust dependency will stay to replace C code with an actual safe language.

You can disable Rust integration in 3.4.x (see FAQ). Starting with 3.5 cryptography will have mandatory Rust code.

[clanzett]

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...
But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

[mgorny]

I guess this means we'll have to fork Cryptography, and render the original package irrelevant.

[tiran]

So the only "reason" for this change is to use a "actual safe language"? ...ahhh...yeah...

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

But blowing up systems with unnecessary software packages is a more secure way, that's for sure...
(my 2 ¢)

The new Rust code adds exactly 0 (zero) runtime packages to Cryptography. Rust, Cargo, pyo3, its dependencies, and setuptools_rust are build-time dependencies only.

[silverfisk]

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

[clanzett]

Memory safety is of course a compelling reason, but you should also keep in mind that may developers out there are using your package, which is great by the way, and are now forced to upgrade all their containers, ci/cd pipelines, packages and so on. A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

[clanzett]

Pinned to second last version to get our pipelines up and running again, at least until I've figured out what should be done long term

Yes, that's exactly what I did.

[Smirl]

I think that it is clear that you have broken semantic versioning with both the python 2 change, and the requirement for rust at build time. From the changelog:

BACKWARDS INCOMPATIBLE: Support for Python 2 has been removed.
We now ship manylinux2014 wheels and no longer ship manylinux1 wheels. Users should upgrade to the latest pip to ensure this doesn’t cause issues downloading wheels on their platform.
cryptography now incorporates Rust code. Users building cryptography themselves will need to have the Rust toolchain installed. Users who use an officially produced wheel will not need to make any changes. The minimum supported Rust version is 1.45.0.
cryptography now has PEP 484 type hints on nearly all of of its public APIs. Users can begin using them to type check their code with mypy.

This should probably be a 4.0 release it would cause a lot less pain for the community. I suppose many have cryptography>=3.0,<4 in setup.py or requirements.txt.

Update
I was unaware that you don't use semantic versioning. I have found in the docs https://cryptography.io/en/latest/api-stability.html#versioning. However it was a little confusing given the style. As many of our dependencies require this we will try not to pin a version and install rust/use manywheel packages.

[tiran]

A little piece of information upfront or a deprecation warning would have helped a lot. Or maybe I just overlooked something.

Looks like you have missed all announcements and FAQ entries.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

PS: I've been preparing Fedora packaging for almost three months. My Fedora Rawhide packages of python-cryptography 3.4 landed about 12 hours after the release -- and I had to jump through several extra hoops because Fedora does not permit vendoring of Rust crates.

[mgorny]

Yes, memory safety is a very compelling reason to use Rust. C is a bad language to implement parsers for e.g. ASN.1. Check out @alex and @reaperhulk side-project https://twitter.com/LazyFishBarrel

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

Rust bindings kicked off in July 2020 with #5357. Alex started a thread on the cryptography developer mailing list in December to get feedback from packagers and users. The FAQ also contains instrutions how you can disable Rust bindings.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

[tiran]

I'm sure all the users who aren't able to use cryptography anymore will appreciate the memory safety.

The majority of users either uses binary wheels (macOS x86_64, glibc Linux x86_64 + aarch64, Windows X86 + X86_64) or Linux distro packages. Binary wheels don't require an additional Rust libraries. Only users on Alpine (musl), BSD, other hardware platforms, and distro packagers are affected.

Do you seriously expect people who maintain a few hundred Python packages to follow development mailing lists of every single one of them?

Do you have constructive suggestions how to communicate changes additionally to project mailing lists, Github issue tracker, project IRC channel, documentation (changelog, FAQ), and Twitter channels?

[public]

@tiran

I wonder if there's a way to produce a more helpful error message, perhaps with a link to some documentation about the new required build deps. The current message most people will hit is

      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed.
      =============================DEBUG ASSISTANCE=============================

Which does sort of attempt to guide people the right way but could be more specific for common cases like Alpine.

Trying to follow the docs as an Alpine user who doesn't know about Rust is kind of confusing right now. As the Alpine specific steps only tell you to install gcc but as you point out above latest Alpine actually has a new enough rust in APK.