net/netip: add new IP address package, use in net

[bradfitz]

I propose we fix #18804 (net: reconsider representation of IP) by importing the inet.af/netaddr into the standard library, probably as net/netaddr, so the IP type is netaddr.IP (the net package already has an IP type). We wouldn't need to import all of it, at least not right away, but I think it'd be nice if we did... the package types work together very well.

This blog post explains the advantages of netaddr.IP over net.IP:

https://tailscale.com/blog/netaddr-new-ip-type-for-go/

Notably, it's small, comparable, and doesn't allocate.

Code is at https://github.com/inetaf/netaddr. Everybody has a Go CLA filed ( inetaf/netaddr#181) and the license is compatible.

Having an allocation-free IP address in the standard library means we'd be able to fix #45886 ("net: add API to receive multiple UDP packets (potentially in one system call)") which ends with #45886 (comment) ("We may want to wait on doing anything here until we figure out what to do with IP addresses, which still allocate."). More generally, this would improve Go's UDP performance, which has never gotten much attention. But with things like QUIC and WireGuard (both UDP-based), Go's alloc-heavy UDP handling is starting to get in the way.

Yes, we'd then have two IP types in the standard library.

As part of this, we'd then add:

• conversion funcs in net to go between them (net would depend on netaddr, not vice-versa)
• the net package internals would switch to netaddr.IP for performance/allocs, converting to net.IP or IPAddr on the edges only.
• a couple new methods would be added (notably to UDPConn) to send/receive UDP packets using netaddr.IP alloc-free (we'd then have ~5 ways to send/recv UDP packets instead of ~4)

/cc @danderson, @josharian, @mdlayher, @crawshaw, @tklauser (co-authors), @neild (co-author for #45886)

[mvdan]

cc @marten-seemann for quic-go

[dotwaffle]

If there's any way we could address the JSON marshalling of CIDR address formats at the same time, that would be very convenient -- rather than having to rely on making custom encoders etc.

I realise that the backwards compatibility guarantee means that's unlikely to be fixed in the standard library version of "net", but if it could be addressed in this proposed new package, it would certainly make the life of network engineers much simpler!

Example: https://play.golang.org/p/3as8Ux9AzJG

[bradfitz]

If there's any way we could address the JSON marshalling of CIDR address formats at the same time, that would be very convenient -- rather than having to rely on making custom encoders etc.

https://play.golang.org/p/5QWiD-tArCc

fmt:	 192.0.2.0/24
json:	 "192.0.2.0/24"

[taktv6]

As someone who had to invent their own IP datatype (https://pkg.go.dev/github.com/bio-routing/bio-rd/net#IP) for the very same reasons, I'm really glad to see progress on this topic. Looking forward to see a decent IP datatype in Golang.

Thanks!

[rsc]

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

[kylelemons]

Would it be worth considering adding a runtime primitive of some sort to help avoid the "workarounds" in the intern package? I don't think it would need to be done at the same time, but having a user for it in the stdlib might make it more compelling.

[Merovius]

@kylelemons There have been and are some proposals related to that, most notable probably #43615. In the interim, the workarounds are much less problematic when they are in the stdlib, because if anything is changed in the implementation that would prevent them from working, they can be changed alongside with it.

[bradfitz]

@kylelemons, what @Merovius said. And to be clear (to anybody just researching the intern package and being horrified), we are definitely not proposing exposing the intern package or any of its API. It's very much an internal implementation detail.

[zephyr]

I don’t want to start bikeshedding, but i think that the proposed type names look a little bit cluttered, since they all share the IP prefix:

net/netaddr.IP
net/netaddr.IPPort
net/netaddr.IPPrefix
net/netaddr.IPRange
net/netaddr.IPSet
net/netaddr.IPSetBuilder

Couldn’t the IP-part just move into the package name instead:

net/ip.Addr
net/ip.Port
net/ip.Prefix
net/ip.Range
net/ip.Set
net/ip.SetBuilder

Or maybe something like net/netip as package name, if net/ip is too short?

Otherwise: Thank you, @bradfitz, for this great proposal and your highly enjoyable blog post!

[mdlayher]

My only nitpick about the package name ip is that I'd guess that is the single most commonly used variable name for any variable of type net.IP, so there's a high likelihood of having to change significant sections of code to avoid shadowing the import net/ip in code being retrofit with the new package APIs.

[zephyr]

@mdlayher While i still think that ip is the nicest option in theory, you have certainly a valid practical concern. Maybe something net/ipa (for ›Internet Protocol address«), to avoid shadowing and jet have something short and sensible?

[andig]

Might also make it ip2 as introduced in Go "2". Or use netip as package if we can forget that it belongs below net for a second.

[rsc]

Opened a discussion at #47323.

[gopherbot]

Change https://golang.org/cl/339309 mentions this issue: net/netaddr: add new IP address package

[rsc]

Over on #47323, people seem generally OK with package net/netip containing

type Addr
type AddrPort
type Prefix
type Range

Commenting here for more visibility. Please comment on #47323 if you have opinions. Thanks.

[rsc]

Given that there seems to be consensus on the four types, it seems like the full API for those would be:

package net

func (r *Resolver) LookupNetAddr(ctx context.Context, network, host string) ([]netip.Addr, error)

func TCPAddrPort(addr netip.AddrPort) *TCPAddr
func (a *TCPAddr) AddrPort() netip.AddrPort

func UDPAddrPort(addr netip.AddrPort) *UDPAddr
func (a *UDPAddr) AddrPort() netip.AddrPort
 
func (c *UDPConn) ReadMsgUDPAddrPort(b, oob []byte) (n, oobn, flags int, addr netip.AddrPort, err error)
func (c *UDPConn) WriteToUDPAddrPort(b []byte, addr netip.AddrPort) (int, error)
func (c *UDPConn) WriteMsgUDPAddrPort(b, oob []byte, addr netip.AddrPort) (n, oobn int, err error)


package netip

type Addr struct { ... unexported ... }

func From4(a,b,c,d byte) Addr
func From16(addr [16]byte) Addr  // "raw", no unmapping
func FromSlice(std []byte) (ip Addr, ok bool)  // "raw", no unmapping
func IPv6LinkLocalAllNodes() Addr
func IPv6Unspecified() Addr
func MustParseAddr(s string) Addr
func ParseAddr(s string) (Addr, error)

func (ip Addr) AppendTo(b []byte) []byte
func (ip Addr) As16() [16]byte
func (ip Addr) As4() [4]byte
func (ip Addr) BitLen() uint8
func (ip Addr) Compare(ip2 Addr) int
func (ip Addr) Is4() bool
func (ip Addr) Is4in6() bool
func (ip Addr) Is6() bool
func (ip Addr) IsGlobalUnicast() bool
func (ip Addr) IsInterfaceLocalMulticast() bool
func (ip Addr) IsLinkLocalMulticast() bool
func (ip Addr) IsLinkLocalUnicast() bool
func (ip Addr) IsLoopback() bool
func (ip Addr) IsMulticast() bool
func (ip Addr) IsPrivate() bool
func (ip Addr) IsUnspecified() bool
func (ip Addr) IsValid() bool
func (ip Addr) IsZero() bool
func (ip Addr) Less(ip2 Addr) bool
func (ip Addr) MarshalBinary() ([]byte, error)
func (ip Addr) MarshalText() ([]byte, error)
func (ip Addr) Next() Addr
func (ip Addr) Prefix(bits uint8) (Prefix, error)
func (ip Addr) Prev() Addr
func (ip Addr) String() string
func (ip Addr) StringExpanded() string
func (ip Addr) Unmap() Addr
func (ip *IP) UnmarshalBinary(b []byte) error
func (ip *IP) UnmarshalText(text []byte) error
func (ip Addr) WithZone(zone string) Addr
func (ip Addr) Zone() string

type AddrPort struct { ... unexported ... }

func AddrPortFrom(ip Addr, port uint16) AddrPort
func ParseAddrPort(s string) (AddrPort, error)
func MustParseAddrPort(s string) AddrPort

func (p AddrPort) AppendTo(b []byte) []byte
func (p AddrPort) Addr() Addr
func (p AddrPort) IsValid() bool
func (p AddrPort) IsZero() bool
func (p AddrPort) MarshalText() ([]byte, error)
func (p AddrPort) Port() uint16
func (p AddrPort) String() string
func (p *AddrPort) UnmarshalText(text []byte) error
func (p AddrPort) WithIP(ip Addr) AddrPort
func (p AddrPort) WithPort(port uint16) AddrPort

type Prefix struct { ... unexported ... }

func PrefixFrom(ip Addr, bits int) Prefix
func MustParsePrefix(s string) Prefix
func ParsePrefix(s string) (Prefix, error)

func (p Prefix) AppendTo(b []byte) []byte
func (p Prefix) Bits() int
func (p Prefix) Contains(ip Addr) bool
func (p Prefix) Addr() Addr
func (p Prefix) IsSingleIP() bool
func (p Prefix) IsValid() bool
func (p Prefix) IsZero() bool
func (p Prefix) MarshalText() ([]byte, error)
func (p Prefix) Overlaps(o Prefix) bool
func (p Prefix) String() string
func (p *Prefix) UnmarshalText(text []byte) error

I've also updated the discussion's top comment with this API. Please comment there if you have opinions. Thanks.

[rsc]

Minor updates at #47323 (comment).

[rsc]

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

[bradfitz]

@rsc, I assume your #46518 (comment) comment is just missing the Range API, as the comment before it includes it as one of the four types?

[rsc]

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

[gopherbot]

Change https://golang.org/cl/360634 mentions this issue: doc/go1.18: add net/netip

[gopherbot]

Change https://golang.org/cl/370134 mentions this issue: doc/go1.18: clarify additions to net package API

[gopherbot]

Change https://golang.org/cl/373834 mentions this issue: doc/go1.18: list new net/netip and net functions and methods