`quinn-udp` should propogate truncation information

[AaronKutch]

In this block here

quinn/quinn-udp/src/unix.rs

Line 550 in 955c1ac

if hdr.msg_flags & libc::MSG_TRUNC != 0 {

, it seems to just silently drop packets when they are truncated. I don't see the same thing elsewhere, I presume it silently truncates in the other implementations, but I would want to know there as well. Hilariously, AI is hallucinating a pub truncated: bool, field in RecvMeta, I'm thinking that is the best approach (I am mainly only using AI to ask about ridiculous OS related things, to be clear).

I know that I could just use large enough byte buffers everywhere, but there are networks with large MTUs and I even forsee possible u16::MAX buffers in my use cases. I would prefer to just detect truncation and write code once to automatically increase buffer sizes as necessary so that I don't have to think about this.

[thomaseizinger]

This code-path seems to only exist for macOS when apple-fast-path feature is not used. I wonder why we only check it there.

I think the easiest to do here would be to return an error (instead of extending RecvMeta. You are ought to try and continue receiving anyway, even on error so that shouldn't break anything and it would mean that this is no longer silently discarded.

[Ralith]

I would prefer to just detect truncation and write code once to automatically increase buffer sizes as necessary so that I don't have to think about this.

This will lead to packet loss, since you can't retry receiving the same packet. Why not size buffers based on interface MTU? You can also advertise the maximum datagram size you're willing to handle at the QUIC layer, if applicable.

[AaronKutch]

I am actually using this crate for something other than QUIC. I think I have enough options to avoid needing to extend RecvMeta if you want to avoid it. However, there is still an issue that truncated packets are passed through on most implementations, I would prefer that the packet is always dropped on truncation, because that can cause a lot of problems for my use case. It should always pass through packets with their true length or not at all. Or it could pass them through but with a stride > len value or something like that so that I can sense truncation.

I feel that there are some missing docs on the basic workings of the sockets and best practices. I was previously using it by passing one large buffer and letting it fill up the buffer with stride < len. However, I should be passing to it a number of buffers equal to the batch size for GRO or GSO, and it should always put one packet per buffer (and I double check stride == len) correct? The previous case would barely take advantage of GRO because the OS would have to redo everytime a different sized packet came, even if from the same address.

[thomaseizinger]

Here is how we are segmenting the buffers to take advantage of GRO: https://github.com/firezone/firezone/blob/3ff31e3a33f923c876e59503ba23497bf6447474/rust/connlib/socket-factory/src/lib.rs#L502-L633

GSO is also non-trivial: https://github.com/firezone/firezone/blob/3ff31e3a33f923c876e59503ba23497bf6447474/rust/connlib/socket-factory/src/lib.rs#L375-L394

Also see #2201.

However, there is still an issue that truncated packets are passed through on most implementations, I would prefer that the packet is always dropped on truncation

I agree with that, we should always return an error here IMO. Wanna send a PR?

[thomaseizinger]

I was previously using it by passing one large buffer and letting it fill up the buffer with stride < len. However, I should be passing to it a number of buffers equal to the batch size for GRO or GSO, and it should always put one packet per buffer (and I double check stride == len) correct?

Ideally you do both (at least on Linux): Pass large buffers so the OS doesn't need to segment for you and pass multiple buffers at the same time so you are not limited to ~65k per syscall.

[Ralith]

However, I should be passing to it a number of buffers equal to the batch size for GRO or GSO, and it should always put one packet per buffer (and I double check stride == len) correct?

This would effectively disable GRO.

The previous case would barely take advantage of GRO because the OS would have to redo everytime a different sized packet came, even if from the same address.

GSO and GRO are only relevant when sending/receiving batches of datagrams with uniform size. If you're not sending flights of max-size datagrams, you're not trying very hard to saturate your link, so reducing CPU overhead shouldn't be a big concern.

[AaronKutch]

I agree with that, we should always return an error here IMO. Wanna send a PR?

I think I or a coworker will make a PR to do this in the coming weeks. Although I think there would be a potential DoS vector if we returned an error for an entire batch, it should drop or signal per packet some way.

[AaronKutch]

This would effectively disable GRO.

Are you sure? In my particular use case, there are reasons for me to put packets in separate buffers because it allow avoiding a copy. I only see a single syscall being made in any case. Or does it cause performance complications on the OS or NIC side that are equivalent to having the overhead of an equal number of syscalls on my side?

I would also like an equivalent of send_to_vectored (but not with a series of slices that are supposed to be interpreted as being continuous, but rather with a separate Transmit for each slice, this is for zerocopy reasons). Is this possible on any platforms?

[mxinden]

However, I should be passing to it a number of buffers equal to the batch size for GRO or GSO, and it should always put one packet per buffer (and I double check stride == len) correct?

This would effectively disable GRO.

Are you sure?

A GRO read reads the entire batch of UDP datagrams (sometimes called segments) into a single contiguous buffer. Along with it, the OS provides the segment size, thus enabling the user to split the larger buffer into individual datagrams.

A GSO write writes the entire batch of UDP datagrams (here too sometimes called segments) from a single contiguous buffer. Along with it, the user provides the segment size, thus enabling the OS to split the larger buffer into individual datagrams.

Servicing multiple buffers is done via the many-message syscalls, e.g. sendmmsg and recvmmsg on Linux.

If you pass multiple buffers, each equal to the segment size, in sum equal to the batch size, the OS can not do GRO (GSO respectively).

Let me know if this helps @AaronKutch.

[thomaseizinger]

Servicing multiple buffers is done via the many-message syscalls, e.g. sendmmsg and recvmmsg on Linux.

quinn-udp used to support those IIRC but I think it got removed because a performance gain could not be measured? See #1729.

[mxinden]

Nit pick:

• Multi-message on the send path is gone since Transmit directly from connection tasks #1729, yes.
• Multi-message on the receive path is still in place. See e.g. for Linux, note the [IoSliceMut<'_>].
  

  quinn/quinn-udp/src/unix.rs

  Lines 464 to 465 in 955c1ac

  	#[cfg(not(any(apple, target_os = "openbsd", target_os = "netbsd", solarish)))]
  	fn recv(io: SockRef<'_>, bufs: &mut [IoSliceMut<'_>], meta: &mut [RecvMeta]) -> io::Result<usize> {