Support for sending multicast packets on multiple interfaces with a single socket

[bltavares]

Hi there @alexcrichton,

Thanks a lot for the project. It has helped me to get started with a multicast-based project, and it was amazing. I was able to send messages and find peers, until I couldn't anymore. I've dig deep into the depths of the network stack, to find, with amusement, that network is chaos.

One of my computers could find the other announcements, but the other computer would not find the first one. After some analyses, I realized this is related to having multiple interfaces, which is very common on laptops (wifi and cable) and Windows machines with HyperV (and WSL2). To my surprise, I could listen to multicast packets on any of the interfaces, and it would only send on a specific one instead of a broadcast to all.

socket2-rs already supports most of the calls to have a "Single socket, multi interface listeners". If we bind to 0.0.0.0, we can use Socket::join_multicast_v4 for each interface ip (enumerating it is outside the package scope).

As we bind to 0.0.0.0, selecting the interface to use for sending the packets is delegated to the OS, and it might end-up selecting the interface that does not talk to the other computers, leading to partial discovery and something that I find very often with my dual network computer.

I found this StackOverflow answer that shows that it would be possible to use the already available Socket::set_multicast_if_v4 call to determine which interface to use before sending the packet, so we can hop around to respond on the same interface that we received the packet.

As you may already know, the default socket API does not return the origin interface on the packet, which made me despaired to find alternatives and question my own sanity of why I'm shaving this yak. I was surprised to find some incantations using IP_PKTINFO and/or/I'm not sure anymore IP_RECVIF to load the information from a "magic" place of the network stack, using CMSG_FIRSTHDR, CMSG_NXTHDR, CMSG_DATA, and WSARecvMsg on Windows.

It looks like this is what Avahi does the interface hopping already, in order to use a single socket to broadcast messages on all interfaces and locate peers happily until the next network topology breaks it again.

My question here; TL; DR; Would it make sense for the socket2-rs project to expose this extra information from CMSG_DATA and WSARecvMsg?

I do think I could make something as a separate crate, but it would also benefit a lot from the already available Windows/Unix infrastructure to build this project on the sys.rs file. If you do think it fits this project, I could try to get something running, but I would also be super ok to ask to have this on another crate.

The alternative seems to be create a socket per interface and put a MultiSocket struct grouping them, which would avoid all this side-quest, but it is not resource efficient, and I'm afraid I would displease Ferris by opening too many sockets.

If you have any other suggestions from your experience, I would love to hear them as well! This is way out of my depth hehehe

Thank you a lot for your attention and time to read this up :)

(PS: I've tried to make this issue funny to read, and I'm sorry ahead of time if it ended up too verbose, or came up weird. This hobby project is leading to so many rabbit holes but I'm learning a lot!)

[alexcrichton]

Thanks for the explanation and the thorough issue report! I suspect that the bindings here are pretty simple so I think they'd work well in this crate. In general this crate is "mostly single-line wrappers around system functionality", so as long as it fits that bill it should be good to add.

[bltavares]

Hi @alexcrichton, thanks for the comment. I've started exploring the design space and doing something usable across platforms required many lines of unsafe code.

I've started to do that on a fork of socket2, but as it kept growing I felt like it would be a bit too big for the project. So I've moved those out to a new crate using some of the OS-specific traits to get access to the socket handle, which was great!

I've put the functionality on multicast-socket, and maybe there are a few things that could be added back here, such as socket.set_pktinfo(true), or exposing to_s_addr, which would help me reduce some copy-and-paste.

The actual sendmsg and recvmsg seems to be a bit more complex to be packaged as cross-patform generic code while trying to be a "single-line wrapper" and I wonder if it would be worth extracting out.

Thanks a lot for providing the AsRawFd and AsRawSocket crate, which allowed to access the underlying handler to wrap things on that nice little crate :) I've tested it on Android, Windows, Linux, Mac and even MIPS and I can do mdns discovery now!

Feel free to either close the issue or suggest any of the code to be contributed back that fits project. Thanks for the attention!

[Thomasdezeeuw]

@bltavares what concretely has to be done to solve this issue? In other words what is socket2 missing (on the master branch)?

[bltavares]

I think we could close the issue, as it is already possible to use the AsRawFd to access things from a socket2::Socket, but there are a few things that could be ported over to socket2, but I'm not sure if that is the intention of the crate.

• Expose to_s_addr to avoid copy-and-paste
• Add socket.set_pkt_info(true) that wraps this piece similarly to how socket.set_reuse_address(true) works
• A more generic way to find socket extensions, such as WSARecvMsgExtension on windows
• A socket.bind_multicast which abstracts the differences between platforms
• Expose recvmsg, and sendmsg (and on windows, the equivalents from an extension method WSA*) to be able to make syscalls with more extensions

This is not required to work, more of a list of ideas when using socket2 on this specific use-case. Given the crate can be used through the AsRawFd trait I would be ok to close this issue and only pick some of the ideas that match the crate goal :)

[Thomasdezeeuw]

I think we could close the issue, as it is already possible to use the AsRawFd to access things from a socket2::Socket, but there are a few things that could be ported over to socket2, but I'm not sure if that is the intention of the crate.

* Expose [to_s_addr](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L443-L449) to avoid copy-and-paste

Is SockAddr::{family, len, as_ptr} sufficient for this?

socket2/src/sockaddr.rs

Lines 65 to 78 in 9c4a404

	/// Returns this address's family.
	pub const fn family(&self) -> sa_family_t {
	self.storage.ss_family
	}
	/// Returns the size of this address in bytes.
	pub const fn len(&self) -> socklen_t {
	self.len
	}
	/// Returns a raw pointer to the address.
	pub const fn as_ptr(&self) -> *const sockaddr {
	&self.storage as *const _ as *const _
	}

* Add `socket.set_pkt_info(true)` that wraps [this piece](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L443-L449) similarly to how `socket.set_reuse_address(true)` works

A pr for this would definitely be accepted.

* A more generic way to find socket extensions, such as [WSARecvMsgExtension](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L150) on windows

* A `socket.bind_multicast` which abstracts [the differences between platforms](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L158-L160)

These would be a bit harder but we can work on it. At the very least we could expose the OS specific functions.

* Expose [recvmsg](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/unix.rs), and [sendmsg](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/unix.rs#L184) (and on windows, the equivalents from [an extension method WSA*](https://github.com/bltavares/multicast-socket/blob/d62ec76551b050364bacf4f9540492279644077c/src/win.rs#L306)) to be able to make syscalls with more extensions

We expose Socket::recv_with_flags:

socket2/src/socket.rs

Lines 318 to 328 in 9c4a404

	/// Identical to [`recv`] but allows for specification of arbitrary flags to
	/// the underlying `recv` call.
	///
	/// [`recv`]: Socket::recv
	pub fn recv_with_flags(
	&self,
	buf: &mut [MaybeUninit<u8>],
	flags: sys::c_int,
	) -> io::Result<usize> {
	sys::recv(self.inner, buf, flags)
	}

And Socket::recv_vectored_with_flags:

socket2/src/socket.rs

Lines 363 to 381 in 9c4a404

	/// Identical to [`recv_vectored`] but allows for specification of arbitrary
	/// flags to the underlying `recvmsg`/`WSARecv` call.
	///
	/// [`recv_vectored`]: Socket::recv_vectored
	///
	/// # Safety
	///
	/// `recv_from_vectored` makes the same safety guarantees regarding `bufs`
	/// as [`recv_vectored`].
	///
	/// [`recv_vectored`]: Socket::recv_vectored
	#[cfg(not(target_os = "redox"))]
	pub fn recv_vectored_with_flags(
	&self,
	bufs: &mut [MaybeUninitSlice<'_>],
	flags: i32,
	) -> io::Result<(usize, RecvFlags)> {
	sys::recv_vectored(self.inner, bufs, flags)
	}

Or do you need access to msghdr structure?

[JonathanPlasse]

• There is a similar crate that do this a-ba/udp_sas.

[Thomasdezeeuw]

We got support for sendmsg and recvmsg and I think we support all required socket options, so closing.