feat!: Add QUIC Address Discovery to iroh (#3049)

## Description
There were two main things to "solve" on the iroh side:
1) gross circular dependency between magicsock and the quinn endpoint
2) needing to handle QAD packets in a special way, like we do with STUN
packets that get sent from net-report

The first was accomplished by figuring out how to make
`MagicSock::spawn` be the location that builds the `quinn::Endpoint`.
This is what was changed:
- implemented `AsyncUdpSocket` on `MagicSock` itself, rather than
`magicsock::Handle`
- `magicsock::Handle` now contains the `quinn::Endpoint`
- we now pass down a `server_config` as a `MagicSock::Option`
- in `MagicSock::spawn`:
- after building the `MagicSock` we now build the `quinn::Endpoint`
using `Arc<MagicSock>` as the `AsyncUdpSocket`
- then we clone the `quinn::Endpoint` and passed it to the
`magicsock::Actor` (which is independent of `MagicSock`)
- give the `quinn::Endpoint` to the `magicsock::Handle`, which is the
actual struct that we use to interact with the magicsocket
- the `iroh::Endpoint` now interacts with the `quinn::Endpoint` using
`msock.endpoint()` in all places

~The second was accomplished by keeping a list of special "QAD
addresses" on the `NodeMap` (`NodeMap::qad_addrs`), and using specific
methods to `add_qad_addrs`, `get_qad_addrs_for_send` and
`get_qad_addrs_for_recv` to deal with the fickle way that the
`quinn::Endpoint` expects SocketAddrs to behave when it dials and when
it receives packets:~
- ~before we do a net-report, we first attempt to resolve the
`RelayUrl`s in the `RelayMap` to get the `SocketAddr`s that we expect we
will dial when we do a QAD probe~
- ~we add those addresses to the `NodeMap` ~
- ~on the "send" side of the `AsyncUdpSocket`, after we do our normal
checks for QuicMappedAddrs, we then check to see if the QuicMappedAddr
is actually just a normal socket addr that we expect to use for QAD. If
so, we just send the packets using the `get_qad_addr_for_send` address~
- ~on the "recv" side of the `AsyncUdpSocket`, after we check to see if
the recv'd address can be mapped to a QuicMappedAddr & therefore can be
received using our normal process, we then check to see if the address
is one that we expect for QAD. If so, we make sure to associate the
packet with the `get_qad_addr_for_recv` address and pass it along~

~The most "unreliable" bits of this are due to dns. Before running a net
report, we now have to do dns discovery for all the RelayUrls. I've
capped this at 300 ms but it means that until we cache the dns responses
then we have this delay before starting net-report. I've also done a bit
of a cheat: when we initially start the `magicsock::Actor`, I've added a
call to `resolve_qad_addrs` that has a timeout of 10 ms, just to send
out the dns packets and hopefully get a jump on caching the responses.~

The second was accomplished by creating a new `IpMappedAddrs` struct
that keeps track of `IpMappedAddrs` to the actual `SocketAddr` that it
represents. This is akin to how we deal with `QuicMappedAddr` (which is
now called `NodeIdMappedAddr`. `netreport` now takes an optional
`IpMappedAddrs`, and when it resolves a `RelayUrl`, it adds the resolved
IP addr to the `IpMappedAddrs`, and uses the mapped address when sending
on QUIC. In `iroh`, we check to see if the destination or source address
is an `IpMappedAddr` or a `NodeIdMappedAddr`. If so, it maps the
addresses correctly before returning the transmit to the endpoint or
sending the transmit over UDP.

## Most interesting bit
So...if someone adds an IP address to `IpMappedAddrs`, we can allow
folks to use the Endpoint as a sort of normal quinn::Endpoint, bypassing
our special iroh holepunching sauce.

depends on #3032 

## Breaking Changes

- `iroh-net-report`
    - changed
- `iroh_net_report::Client::new` now takes additional parameter
`mapped_addrs: `Option<IpMappedAddrs>`

## Change checklist

- [x] Self-review.
- [x] Documentation updates following the [style
guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text),
if relevant.
- [x] Tests if relevant.
- [ ] All breaking changes documented.

---------

Co-authored-by: “ramfox” <“kasey@n0.computer”>
Co-authored-by: Floris Bruynooghe <flub@n0.computer>
Co-authored-by: Asmir Avdicevic <asmir.avdicevic64@gmail.com>

iroh-net-report/src/ip_mapped_addrs.rs

@@ -0,0 +1,131 @@
+use std::{
+    collections::BTreeMap,
+    net::{IpAddr, Ipv6Addr, SocketAddr},
+    sync::{
+        atomic::{AtomicU64, Ordering},
+        Arc,
+    },
+};
+
+/// The dummy port used for all mapped addresses
+pub const MAPPED_ADDR_PORT: u16 = 12345;
+
+/// Can occur when converting a [`SocketAddr`] to an [`IpMappedAddr`]
+#[derive(Debug, thiserror::Error)]
+#[error("Failed to convert")]
+pub struct IpMappedAddrError;
+
+/// A map fake Ipv6 address with an actual IP address.
+///
+/// It is essentially a lookup key for an IP that iroh's magicsocket knows about.
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash, Ord, PartialOrd)]
+pub struct IpMappedAddr(Ipv6Addr);
+
+/// Counter to always generate unique addresses for [`IpMappedAddr`].
+static IP_ADDR_COUNTER: AtomicU64 = AtomicU64::new(1);
+
+impl IpMappedAddr {
+    /// The Prefix/L of our Unique Local Addresses.
+    const ADDR_PREFIXL: u8 = 0xfd;
+    /// The Global ID used in our Unique Local Addresses.
+    const ADDR_GLOBAL_ID: [u8; 5] = [21, 7, 10, 81, 11];
+    /// The Subnet ID used in our Unique Local Addresses.
+    const ADDR_SUBNET: [u8; 2] = [0, 1];
+
+    /// Generates a globally unique fake UDP address.
+    ///
+    /// This generates a new IPv6 address in the Unique Local Address range (RFC 4193)
+    /// which is recognised by iroh as an IP mapped address.
+    pub fn generate() -> Self {
+        let mut addr = [0u8; 16];
+        addr[0] = Self::ADDR_PREFIXL;
+        addr[1..6].copy_from_slice(&Self::ADDR_GLOBAL_ID);
+        addr[6..8].copy_from_slice(&Self::ADDR_SUBNET);
+
+        let counter = IP_ADDR_COUNTER.fetch_add(1, Ordering::Relaxed);
+        addr[8..16].copy_from_slice(&counter.to_be_bytes());
+
+        Self(Ipv6Addr::from(addr))
+    }
+
+    /// Return a [`SocketAddr`] from the [`IpMappedAddr`].
+    pub fn socket_addr(&self) -> SocketAddr {
+        SocketAddr::new(IpAddr::from(self.0), MAPPED_ADDR_PORT)
+    }
+}
+
+impl TryFrom<Ipv6Addr> for IpMappedAddr {
+    type Error = IpMappedAddrError;
+
+    fn try_from(value: Ipv6Addr) -> std::result::Result<Self, Self::Error> {
+        let octets = value.octets();
+        if octets[0] == Self::ADDR_PREFIXL
+            && octets[1..6] == Self::ADDR_GLOBAL_ID
+            && octets[6..8] == Self::ADDR_SUBNET
+        {
+            return Ok(Self(value));
+        }
+        Err(IpMappedAddrError)
+    }
+}
+
+impl std::fmt::Display for IpMappedAddr {
+    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
+        write!(f, "IpMappedAddr({})", self.0)
+    }
+}
+
+/// A Map of [`IpMappedAddresses`] to [`SocketAddr`].
+// TODO(ramfox): before this is ready to be used beyond QAD, we should add
+// mechanisms for keeping track of "aliveness" and pruning address, as we do
+// with the `NodeMap`
+#[derive(Debug, Clone, Default)]
+pub struct IpMappedAddresses(Arc<std::sync::Mutex<Inner>>);
+
+#[derive(Debug, Default)]
+pub struct Inner {
+    by_mapped_addr: BTreeMap<IpMappedAddr, SocketAddr>,
+    /// Because [`std::net::SocketAddrV6`] contains extra fields besides the IP
+    /// address and port (ie, flow_info and scope_id), the a [`std::net::SocketAddrV6`]
+    /// with the same IP addr and port might Hash to something different.
+    /// So to get a hashable key for the map, we are using `(IpAddr, u6)`.
+    by_ip_port: BTreeMap<(IpAddr, u16), IpMappedAddr>,
+}
+
+impl IpMappedAddresses {
+    /// Creates an empty [`IpMappedAddresses`].
+    pub fn new() -> Self {
+        Self(Arc::new(std::sync::Mutex::new(Inner::default())))
+    }
+
+    /// Adds a [`SocketAddr`] to the map and returns the generated [`IpMappedAddr`].
+    ///
+    /// If this [`SocketAddr`] already exists in the map, it returns its
+    /// associated [`IpMappedAddr`].
+    ///
+    /// Otherwise a new [`IpMappedAddr`] is generated for it and returned.
+    pub fn get_or_register(&self, socket_addr: SocketAddr) -> IpMappedAddr {
+        let ip_port = (socket_addr.ip(), socket_addr.port());
+        let mut inner = self.0.lock().expect("poisoned");
+        if let Some(mapped_addr) = inner.by_ip_port.get(&ip_port) {
+            return *mapped_addr;
+        }
+        let ip_mapped_addr = IpMappedAddr::generate();
+        inner.by_mapped_addr.insert(ip_mapped_addr, socket_addr);
+        inner.by_ip_port.insert(ip_port, ip_mapped_addr);
+        ip_mapped_addr
+    }
+
+    /// Returns the [`IpMappedAddr`] for the given [`SocketAddr`].
+    pub fn get_mapped_addr(&self, socket_addr: &SocketAddr) -> Option<IpMappedAddr> {
+        let ip_port = (socket_addr.ip(), socket_addr.port());
+        let inner = self.0.lock().expect("poisoned");
+        inner.by_ip_port.get(&ip_port).copied()
+    }
+
+    /// Returns the [`SocketAddr`] for the given [`IpMappedAddr`].
+    pub fn get_ip_addr(&self, mapped_addr: &IpMappedAddr) -> Option<SocketAddr> {
+        let inner = self.0.lock().expect("poisoned");
+        inner.by_mapped_addr.get(mapped_addr).copied()
+    }
+}

iroh-net-report/src/lib.rs

@@ -33,10 +33,12 @@ use tracing::{debug, error, info_span, trace, warn, Instrument};
 
 mod defaults;
 mod dns;
+mod ip_mapped_addrs;
 mod metrics;
 mod ping;
 mod reportgen;
 
+pub use ip_mapped_addrs::{IpMappedAddr, IpMappedAddrError, IpMappedAddresses, MAPPED_ADDR_PORT};
 pub use metrics::Metrics;
 use reportgen::ProbeProto;
 pub use reportgen::QuicConfig;
@@ -348,8 +350,12 @@ impl Client {
     ///
     /// This starts a connected actor in the background.  Once the client is dropped it will
     /// stop running.
-    pub fn new(port_mapper: Option<portmapper::Client>, dns_resolver: DnsResolver) -> Result<Self> {
-        let mut actor = Actor::new(port_mapper, dns_resolver)?;
+    pub fn new(
+        port_mapper: Option<portmapper::Client>,
+        dns_resolver: DnsResolver,
+        ip_mapped_addrs: Option<IpMappedAddresses>,
+    ) -> Result<Self> {
+        let mut actor = Actor::new(port_mapper, dns_resolver, ip_mapped_addrs)?;
         let addr = actor.addr();
         let task = task::spawn(
             async move { actor.run().await }.instrument(info_span!("net_report.actor")),
@@ -566,14 +572,21 @@ struct Actor {
 
     /// The DNS resolver to use for probes that need to perform DNS lookups
     dns_resolver: DnsResolver,
+
+    /// The [`IpMappedAddresses`] that allows you to do QAD in iroh
+    ip_mapped_addrs: Option<IpMappedAddresses>,
 }
 
 impl Actor {
     /// Creates a new actor.
     ///
     /// This does not start the actor, see [`Actor::run`] for this.  You should not
     /// normally create this directly but rather create a [`Client`].
-    fn new(port_mapper: Option<portmapper::Client>, dns_resolver: DnsResolver) -> Result<Self> {
+    fn new(
+        port_mapper: Option<portmapper::Client>,
+        dns_resolver: DnsResolver,
+        ip_mapped_addrs: Option<IpMappedAddresses>,
+    ) -> Result<Self> {
         // TODO: consider an instrumented flume channel so we have metrics.
         let (sender, receiver) = mpsc::channel(32);
         Ok(Self {
@@ -584,6 +597,7 @@ impl Actor {
             in_flight_stun_requests: Default::default(),
             current_report_run: None,
             dns_resolver,
+            ip_mapped_addrs,
         })
     }
 
@@ -644,6 +658,7 @@ impl Actor {
             quic_config,
             ..
         } = opts;
+        trace!("Attempting probes for protocols {protocols:#?}");
         if self.current_report_run.is_some() {
             response_tx
                 .send(Err(anyhow!(
@@ -686,6 +701,7 @@ impl Actor {
             quic_config,
             self.dns_resolver.clone(),
             protocols,
+            self.ip_mapped_addrs.clone(),
         );
 
         self.current_report_run = Some(ReportRun {
@@ -1133,7 +1149,7 @@ mod tests {
             stun_utils::serve("127.0.0.1".parse().unwrap()).await?;
 
         let resolver = crate::dns::tests::resolver();
-        let mut client = Client::new(None, resolver.clone())?;
+        let mut client = Client::new(None, resolver.clone(), None)?;
         let dm = stun_utils::relay_map_of([stun_addr].into_iter());
 
         // Note that the ProbePlan will change with each iteration.
@@ -1181,7 +1197,7 @@ mod tests {
 
         // Now create a client and generate a report.
         let resolver = crate::dns::tests::resolver();
-        let mut client = Client::new(None, resolver.clone())?;
+        let mut client = Client::new(None, resolver.clone(), None)?;
 
         let r = client.get_report(dm, None, None, None).await?;
         let mut r: Report = (*r).clone();
@@ -1384,7 +1400,7 @@ mod tests {
         let resolver = crate::dns::tests::resolver();
         for mut tt in tests {
             println!("test: {}", tt.name);
-            let mut actor = Actor::new(None, resolver.clone()).unwrap();
+            let mut actor = Actor::new(None, resolver.clone(), None).unwrap();
             for s in &mut tt.steps {
                 // trigger the timer
                 tokio::time::advance(Duration::from_secs(s.after)).await;
@@ -1419,7 +1435,7 @@ mod tests {
         dbg!(&dm);
 
         let resolver = crate::dns::tests::resolver().clone();
-        let mut client = Client::new(None, resolver)?;
+        let mut client = Client::new(None, resolver, None)?;
 
         // Set up an external socket to send STUN requests from, this will be discovered as
         // our public socket address by STUN.  We send back any packets received on this