Tracking issue for QUIC support

[kpp]

There are a couple of reasons to support QUIC network protocol.

The first, slightly minor, is that validators are maintaining more than 1000 TCP connections open and are running into file descriptor limits.
Second is we use Yamux for multiplexing multiple substreams into one, which is a rather poor protocol that does the best it can on top of TCP, but can't do more.
Because we can't know the TCP window size from user space, we can't actually properly allocate window sizes to individual substreams. If two or more substreams send a big volume of data, we'll run into head of line blocking issues (This is the same reason why HTTP2 is not so great and is being replaced with HTTP3, if you want to read more)
Third reason is we're suffering from a TCP slow start issue. A TCP connection is almost unused, and suddenly we send for example 2 MiB on it. Because of TCP slow start, it's going to take a rather long time to send these 2 MiBs even if the connection would be capable of sending them quickly.

First attempt was made in paritytech/substrate#6366.

So far there is https://crates.io/crates/libp2p-quic which should be reviewed.

The list of sub issues will be updated time to time:

• TLS 1.3 vs Noise. The golang version uses TLS (see https://github.com/libp2p/specs/blob/master/tls/tls.md#peer-authentication), also there is no nQUIC specs, so to be compatible with libp2p we have to use TLS+QUIC.
• Inspect whether we can replace x509-signature with picky. The answer is we can replace it with any crate we can including x509-parser, picky, etc...
• Implement x509 cert serializer, parser, verifier Add initial code handling certificate kpp/rust-libp2p#1
• Implement verification of all signature algorithms according to TLS 1.3 standard Support all verification algorithms kpp/rust-libp2p#4
• Implement rustls ServerCertVerifier & ClientCertVerifier Implement rustls ServerCertVerifier & ClientCertVerifier kpp/rust-libp2p#6
• Make libp2p-quic binary compatible with the go implementation (a standalone issue) Make tls cert binary compat with the go implementation ipfs-rust/libp2p-quic#6
• R&D: a basic QUIC server R&D: accept tls handshake and read a simple QUIC message kpp/rust-libp2p#8
• The existing crate libp2p-quic now supports TLS, but there still some issues left.
• Integrate libp2p-quic into rust-libp2p with the help of its maintainers Libp2p quic second attempt libp2p/rust-libp2p#2159 transports/quic: Add implementation based on quinn-proto libp2p/rust-libp2p#2289
• Integrate libp2p-quic into substrate Add experimental support for QUIC substrate#11514.

[burdges]

Yes. QUIC would improve our networking significantly. :)

We previously got hung up on keeping Noise IK via nQUIC = QUIC + Noise vs adopting QUIC with TLS 1.3 by defining some new grandpa TLS certificate for validators, crazier certificates for collators, and maybe others.

I'm completely happy outsourcing an initial spec for one or both of these. It's also fine if we do something fast and warn everyone to expect yet another network protocol migration. It's conversely also possible that @tomaka knows some wish-like of libp2p messes that would be good to bundle in with this big a migration.

I'd expect a "grandpa TLS certificate" for validators to consist of a Merkle proof identifying the Ed25519 grandpa session keys, and then a certificate by the session key on the transport key. We should first migrate storage from our radix 16 hashing, which bloats in a dense Merkle tree, to radix 2 hashing with hash fast forwarding (and radix 16 caching). We'll want crazy certificates for collators, like VRFs for parachains running Babe and worse for Sassafras in future.

I've no idea if we properly exploit Noise IK yet anyways, meaning whether nodes initiating connections identify themselves first via something vaguely like this "grandpa TLS certificate". It's annoying if the Merkle proof that shows your actually a validator does not fit into the MTU used for the first datagram, so that 4x space cost of our radix 16 hashing matters here too.

In brief, we need to decide how quickly we want this and what we want to try to rush to merge into such a migration. We'll surely punt on doing authentication perfectly optimal initially, but we're slowly creating the possibility for better authentication elsewhere so it'll circle back here one day. :)

[burdges]

I think the simplest would be QUIC with TLS 1.3 and no authentication, but authentication done late inside the stream. We simultaneously give someone reputable a grant for designing nQUIC properly. And push our storage folk to fix our radix 16 hashing issue.

[tomaka]

I'd strongly prefer for us to implement something that is already specified (i.e. TLS+QUIC) and not start creating parity-nquic that will get abandoned after 6 months.

[kpp]

So far I did a research and implemented:

1. a go tool to generate an x509 certificate with libp2p extension in DER format (the code is mostly a copypasta from https://github.com/libp2p/go-libp2p-tls);
2. a rust tool to parse, inspect and verify the signature according to libp2p docs.

The reason I did it was I wanted to be compatible with the go implementation of libp2p-quic. In the future I want to test our client with their server and vice versa.

https://gist.github.com/kpp/c9c84411e17f4b27dddf0d438b289862

The code is ugly but it's not the point. The point is that the rust tool is binary compatible.

The next step is to prettify the code, implement a certificate serializer and create a PR.

Also I found a tool in the Internets to read DER files which helped me a lot to inspect the certificate: https://lapo.it/asn1js.

[kpp]

So far I:

• updated tool to generate certs: https://gist.github.com/kpp/c9c84411e17f4b27dddf0d438b289862
• implemented verification of certs with rsa-pkc1, ecdsa, ed25519, rsa-pss : Support all verification algorithms kpp/rust-libp2p#4

[kpp]

These two weeks I:

• Worked on Implement rustls ServerCertVerifier & ClientCertVerifier kpp/rust-libp2p#6
• Tried to figure out how to work with quinn_proto: R&D: accept tls handshake and read a simple QUIC message kpp/rust-libp2p#8
• As a standalone issue applied a fix to the libp2p-quic crate: Make tls cert binary compat with the go implementation ipfs-rust/libp2p-quic#6
• Found an issue in the libp2p tls specs: Fix tls.md libp2p/specs#357

[kpp]

While I was playing with quinn_proto, David added TLS support into his libp2p-quic crate. We had a chat with Pierre and came to a conclusion the code is a good starting point. There are some issues left but so far it looks pretty good.

Since the last time I worked on multiple issues:

• Made a code review of libp2p-quic
• Fixed libp2p-quic: Make tls cert binary compat with the go implementation ipfs-rust/libp2p-quic#6
• Worked on specs: Fix tls docs according to specs kpp/rust-libp2p#10, Define precisely how to marshal SubjectPublicKeyInfo structure in tls.md libp2p/specs#358
• Found several issues in the go implementation: Support critical libp2p extension libp2p/go-libp2p-tls#87 Update go-libp2p-tls to 0.2.0 libp2p/go-libp2p-quic-transport#226
• Improved x509 parser: Add SubjectPublicKeyInfo::raw field rusticata/x509-parser#100
• Worked on extracting UDP code into a separate code (Extract udp crate quinn-rs/quinn#1180 quinn_udp: use async_io instead of tokio quinn-rs/quinn#1183) which will help libp2p-quic to reuse the code from quinn.

The current progress is: we are integrating libp2p-quic into rust-libp2p: libp2p/rust-libp2p#2159.

[kpp]

dvc94ch dropped, so I opened my own PR: libp2p/rust-libp2p#2289

• A lot of fixes being made:
• it was requested to remove noise support libp2p/rust-libp2p@25ee1fe
• generic crypto code was removed: libp2p/rust-libp2p@f35fde1
• we can dial without PK: libp2p/rust-libp2p@3b1ff34
• I replaced barebones-x509 to deal with X509 certs: libp2p/rust-libp2p@63e4696
• added some basic QUIC support to libp2p-perf: Actually run quic benches mxinden/libp2p-perf#56

Here are bench results:

Bench results

Local-Local TCP

# Start Rust and Golang servers.

# Rust -> Rust

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.00 s	5913 MBytes	4730.32 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.00 s	9203 MBytes	7362.40 MBit/s

# Rust -> Golang

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.00 s	5458 MBytes	4366.22 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.00 s	10284 MBytes	8227.13 MBit/s

# Golang -> Rust

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.00 s 	6880 MBytes	5502.57 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.00 s 	17534 MBytes	14026.69 MBit/s

# Golang -> Golang

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.00 s 	4881 MBytes	3904.79 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.00 s 	23115 MBytes	18489.50 MBit/s


Local-Local QUIC:

# Start Rust and Golang servers.

# Rust -> Rust

## Transport security noise
Local peer id: PeerId("12D3KooWAEgNvJB6tXtmpgjf2GDZjMhCneZvtUo2KTR4if4kkoH7")
Interval	Transfer	Bandwidth
0 s - 10.00 s	361 MBytes	288.72 MBit/s

## Transport security plaintext
Local peer id: PeerId("12D3KooWHiYCM8HrETLwqDwAwo3ovHWgAo1astGG2bLJ7VdydA95")
Interval	Transfer	Bandwidth
0 s - 10.00 s	381 MBytes	304.77 MBit/s

# Golang -> Golang

## Transport security noise
2021/12/09 14:35:18 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.00 s 	725 MBytes	579.98 MBit/s

## Transport security plaintext
2021/12/09 14:35:28 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.00 s 	724 MBytes	579.09 MBit/s


Local-Server TCP (mixed plaintext/noise):

# Rust -> Rust

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.02 s	126 MBytes	100.64 MBit/s

## Transport security plaintext
Interval	Transfer	Bandwidth
0 s - 10.09 s	125 MBytes	99.08 MBit/s


# Rust -> Golang

## Transport security noise
Interval	Transfer	Bandwidth
0 s - 10.21 s	111 MBytes	86.97 MBit/s

# Golang -> Rust

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.03 s 	129 MBytes	102.92 MBit/s

## Transport security plaintext
Interval 	Transfer	Bandwidth
0s - 10.01 s 	125 MBytes	99.85 MBit/s

# Golang -> Golang

## Transport security noise
Interval 	Transfer	Bandwidth
0s - 10.15 s 	89 MBytes	70.15 MBit/s



Local-Server QUIC:

# Rust -> Rust

## Transport security noise
Local peer id: PeerId("12D3KooWR1KRW9UoJd8XXwFLKWdvmC4yTEDgStvXJGUD3ZFnFZDw")
Interval	Transfer	Bandwidth
0 s - 10.01 s	9 MBytes	7.19 MBit/s

## Transport security plaintext
Local peer id: PeerId("12D3KooWA7vqQpUV3SxUNTWWeWHiM2N1WEebMPETwhLLtmD6QdnL")
Interval	Transfer	Bandwidth
0 s - 10.04 s	4 MBytes	3.19 MBit/s


# Golang -> Golang

## Transport security noise
2021/12/09 17:42:04 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.01 s 	109 MBytes	87.12 MBit/s

## Transport security plaintext
2021/12/09 17:42:14 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Interval 	Transfer	Bandwidth
0s - 10.01 s 	121 MBytes	96.71 MBit/s

Also we still cannot connect to go-libp2p-quic. I am working on it.

[burdges]

It's QUIC with QUIC's standard TLS 1.3 then? Cool we reinvent enough around here anyways. ;)

[kpp]

I discovered that I pollute Swarm threads with StreamMuxer::poll_event with:

let span = tracing::span!(tracing::Level::TRACE, "handle_event").entered();
while let Poll::Ready(event) = inner.endpoint.poll_channel_events(cx) {
    inner.connection.handle_event(event);
}
drop(span);

let span = tracing::span!(tracing::Level::TRACE, "send_transmit").entered();
let max_datagrams = inner.endpoint.max_datagrams();
while let Some(transmit) = inner.connection.poll_transmit(now, max_datagrams) {
    inner.endpoint.send_transmit(transmit);
}
drop(span);

And they are responsible for parsing packets and building packets respectively. I believe that's the main issue for poor performance.
tracing.folded.txt

[kpp]

The current state of this issue:

• We got a robust implementation in transports/quic: Add implementation based on quinn-proto libp2p/rust-libp2p#2289
• There is a tool to bench the performance Add support for QUIC mxinden/libp2p-perf#74
• I am working on integration: Add experimental support for QUIC substrate#11514