Interfacing with Rust

[fjarri]

Refactor to achieve binary compatibility (and API uniformness) with rust-umbral (see the corresponding PR nucypher/rust-umbral#41):

• drop support of Py 3.5
• remove all hashing algorithms but SHA256
• implement unsafe_hash_to_point() in the same way as rust-umbral
• implement hash_to_curvebn() in the same way RustCrypto does it
• use XChaCha instead of Salsa to store encrypted secret keys
• implement adding DST to hashes in the same way as rust-umbral
• use hashing with DST everywhere, with the same tags as rust-umbral
• require metadata on cfrag verification
• add compatibility tests checking cross-operation with Rust
• change API to make it a drop-in replacement for rust-umbral
• remove unused public API
• add API documentation

For reviewers:

The size may seem daunting, but a lot of it is due to vector changes. Perhaps it will be better to review commit-by-commit, or just file by file. The size of both tests and the library itself shrunk by 30-40%, and 25% of the library is just OpenSSL wrapper functions.

I'm planning to make a test PR to nucypher to see if this new version has enough functionality for it. There may be slight changes in the API because of that, but the bulk of the changes is already made.

In particular, BytestringSplitter may have to be brought back. I removed it temporarily because I couldn't figure out splitter composition, and didn't want that to block me.

Performance:

There are two differences between this version and the main branch currently.

1. generate_kfrags is ~15% slower. This is caused by one secret key -> public key computation, which is cached in the main branch. Fixed, but in Python version only - just to avoid explaining why there's a performance regression. It starts getting less and less important the higher m and n you take.
2. reencrypt is ~2x faster. The main branch verifies the capsule twice (4 curve point * scalar multiplications), and kfrag once (1 curve point * scalar multiplication). This effectively doubles the number of multiplications, which are the main bottleneck in this operation. In this PR, capsules are verified on deserialization only, and reencrypt() does not attempt to verify kfrags (the caller must call verification explicitly).

[cygnusv]

In the past, we've opened PRs to cryptography.io to expose such endpoints

pyca/cryptography#4118

[cygnusv]

Usually the term "private" key implicitly signals an asymmetric cryptography scheme (hinting at the private vs public dichotomy), while "secret" key is usually associated to symmetric cryptography.

[fjarri]

I was following the name convention from rust-umbral, which, in turn, follows the terms used by the RustCrypto stack. Is using "secret" instead of "private" here very confusing, or just somewhat unusual?

As an extension of uniformity with RustCrypto, I was thinking of just re-exporting the RustCrypto key objects themselves instead of wrapping them in our own types. This way the keys can be used with other libraries based on RustCrypto, making it unnecessary to have conversion methods like to_cryptography_privkey() we have in PyUmbral.

Also being able to abbreviate to "sk" and "pk" is quite useful.

[cygnusv]

This is an awesome idea 🤯

[cygnusv]

Thanks for the walkthrough yesterday! The PR looks great and to me it's a relief seeing this simplification in so many aspects of Umbral.

I think I gave you my main comments during the walkthrough and I see you are working on that in part 2. I still think the use of "SecretKey" is not the best choice taking into consideration the uses in the industry and academia for public-key cryptosystems. As a random example, look at the recent draft for Hybrid Public Key Encryption from the IETF (https://www.ietf.org/archive/id/draft-irtf-cfrg-hpke-08.html#name-notation); they use the term "private key" even though they use the sk abbreviation.

But that's something that's not critical at all in the scope of this PR.

[cygnusv]

Yay!

[cygnusv]

good thing we're keeping this fixture

[fjarri]

I think I gave you my main comments during the walkthrough and I see you are working on that in part 2. I still think the use of "SecretKey" is not the best choice taking into consideration the uses in the industry and academia for public-key cryptosystems.

Perhaps. Let us put a pin in that and revisit it after existing PRs — changing it in 4 PRs now will lead to horrible rebases.

I agree with following the existing conventions, but I wonder why RustCrypto chose to call it SecretKey - the author is usually very diligent in implementing crypto standards.