Feedback on adopting Rust code

[alex]

We are looking into including Rust code in pyca/cryptography (see #5357). This will be done using PyO3, and thus it requires #5359 to be completed before it will happen. This means this will not happen until 2021 in all likelihood.

We are interested in feedback in things we can do to make this a smooth process, particularly from folks who are re-distributing cryptography. Please note: feedback of the form "don't use Rust" is not productive and will be disregarded.

Here are things we're currently planning to minimize pain:

• Support the stable Rust channel.
• Declare a Minimum Supported Rust Version (MSRV) and test and document it.
• Continue building binary wheels for all the platforms we currently do.

Ideas we're considering:

• Ensuring our MSRV is present in the Debian repositories

[alex]

cc: @glyph @mattsb42-aws @bmw @ohemorange @Lukasa @bitprophet for our various high-profile downstreams

[Lukasa]

I don’t think I’m prominently using cryptography at this time, so...+1, go for it!

[glyph]

The only way I could be happier about this is if it meant you were also eliminating OpenSSL.

[glyph]

(I am slightly concerned about the build-dependencies side of this; particularly that pypy doesn't get binary wheels, and there are already enough ways that this can go wrong in confusing ways today.)

[alex]

Is there some action we could be taking to make building PyPy wheels practical, or is the blocker there defining something akin to abi3 for PyPy?

[geofft]

Can I request that you include some optional Rust, enabled by default but disableable, for a bit of time before you depend on it?

At $work we're rebuilding cryptography from source in our monorepo using our own build system, which calls out to setup.py but manages dependencies on its own (to make sure we're not using system Python, system gcc, system rustc, etc.). We recently had a request for another PyO3 package, which we managed to put together somehow, but for cryptography I'd like to be very sure that it's building properly and using the right dependencies. It sounds from #5357 like you plan to start small - if you can keep the old implementation present and behind a setup.py flag, or something, that would let us both make sure that our build system works with setuptools-rust (or whatever you end up using) and also would give us a temporary option to keep upgrading if it doesn't work yet.

Also you probably want to ping your Debian and Fedora packagers if you haven't.

[alex]

How about this:

• The first release containing PyO3 will actually just build a no-op module. It won't export anything, and won't be used anywhere. But it will be built by setup.py, which should let folks properly test their build infra.
• For the first release only, there'll be an env var you can set which will cause it to not build that module. In other words, you get one prod release to get your build system in order.
• The release after that will contain non-optional Rust code which will need to be built.

[geofft]

Sounds good, and one release cycle sounds like the right amount of time.

I'd ask if you can actually use the module just to make sure we've got dynamic linking working right, but that might be catering too much to people with custom build systems :)

Oh, also - does Anaconda have any PyO3 stuff yet?

[glyph]

I'm honestly more concerned with adoption by new projects than build-system updates for existing ones. Whatever wacky gyrations you require of Twisted, we'll figure it out. But what is the experience of pip install cryptography trying out twisted on pypy3 just to see how it works, for someone with no rust toolchain installed? How do they find out what they're missing? How much configuration is required?

This situation already sucks for people missing fragments of the requisite C compiler toolchain and libraries, but at least there's a ton of folk knowledge well-diffused throughout the community to deal with it. This is going to be both horrible and novel, and it would be great to address that head-on before it ships.

[alex]

The error is pretty clear: error: Can not find Rust compiler. Any of the standard instructions for installing Rust (whether that's the installer from rust-lang.org, brew, etc.) will install Rust in a way that will satisfy the error.

[h-vetinari]

@geofft: Oh, also - does Anaconda have any PyO3 stuff yet?

Not as far as I can tell, but I opened an issue to add it. The rust-compiler is already there, so it should hopefully not take too much effort.

[alex]

Can you link that issue please?

…

On Sun, Aug 9, 2020 at 6:20 PM h-vetinari ***@***.***> wrote: @geofft <https://github.com/geofft>: Oh, also - does Anaconda have any PyO3 stuff yet? Not as far as I can tell, but I opened an issue to add it. The rust-compiler is already there, so it should hopefully not take too much effort. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5381 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBFBFEIG4OMQNMEITRTR74OKZANCNFSM4PYEIJ3Q> .

-- All that is necessary for evil to succeed is for good people to do nothing.