socketserver.TCPServer can not listen IPv6 address

[dazhaoyu]

BPO	20215
Nosy	@gpshead, @jaraco, @nirs, @giampaolo, @bitdancer, @berkerpeksag, @vadmium, @jleedev
PRs	bpo-38907: In http.server script, restore binding to IPv4 on Windows. #17851
Files	issue20215_socketserver.patch: Patch against latest Python 3.5 mercurial branch issue20215-gps01.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-01-10.02:03:02.342>
labels = ['type-feature', 'library']
title = 'socketserver.TCPServer can not listen IPv6 address'
updated_at = <Date 2020-01-06.02:11:41.352>
user = 'https://bugs.python.org/dazhaoyu'

bugs.python.org fields:

activity = <Date 2020-01-06.02:11:41.352>
actor = 'jaraco'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2014-01-10.02:03:02.342>
creator = 'dazhaoyu'
dependencies = []
files = ['35147', '43983']
hgrepos = []
issue_num = 20215
keywords = ['patch']
message_count = 14.0
messages = ['207818', '207824', '217833', '226219', '231363', '231377', '231379', '243884', '271865', '271869', '271871', '271872', '271873', '325416']
nosy_count = 14.0
nosy_names = ['gregory.p.smith', 'jaraco', 'nirs', 'giampaolo.rodola', 'r.david.murray', 'neologix', 'berker.peksag', 'martin.panter', 'jpokorny', 'jleedev', 'dazhaoyu', 'andreasr', 'Carlos.Ralli', 'Paul Marks']
pr_nums = ['17851']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20215'
versions = ['Python 3.5']

[dazhaoyu]

I see, in python 2.7, in SocketServer.py the TCPServer implementation is hard-coded to use ipv4, can not handle IPv6 case. it hard-coded set address_family as socket.AF_INET. so when binding IPv6 host, it will throw "gaierror: [Errno -9] Address family for hostname not supported".

The code should to judge the provided host is IPv4 or IPv6, and base on the host type to set address_family as socket.AF_INET or socket.AF_INET6

[bitdancer]

The *default* is AF_INET. If you are requesting a more convenient API than subclassing (which for IPV6 support I think would be a reasonable request), that would be a new feature.

[andreasr]

Here's a patch which implements support in SocketServer to set IPv6 socket address family to AF_INET6 automatically if the TCPServer is specified to listen to an IPv6 address. This means that users of the TCPServer class will get the correct address family automatically for both IPv4 and IPv6 based on the IP address they specify, and hopefully this will be more userfriendly. The patch is against the latest Python 3.5.

Note that this is my first patch to the Python project, I hope I did everything correct.

[andreasr]

Is there any interest in this patch? it would be nice with a review of the patch. :)

[CarlosRalli]

Is it possible to use this patch for python2.7 ?

[CarlosRalli]

Sorry, I've got nothing. Did you forget the attachment or mail body ?

2014-11-19 11:08 GMT+01:00 Berker Peksag <report@bugs.python.org>:

Changes by Berker Peksag <berker.peksag@gmail.com>:

----------
nosy: +berker.peksag
stage: -> patch review

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue20215\>

---

[bitdancer]

Andreas: I would prefer to avoid a dependency on the ipaddress module. I would suggest adding an address_family constructor argument that defaults to None, where a value of None would mean "just pass the server_address to bind (or getaddrinfo?) and find out what the resulting family is". I'm not exactly a socket programming expert, though, so there may be a reason I'm not aware of that that won't work well.

Carlos: no, this is a new feature, so it can go into 3.5 only at this point.

[vadmium]

I’m no IPv6 expert, but see Lib/smtpd.py:657 and bpo-14758 for how this sort of thing was done for the SMTP server using getaddrinfo(). I think it is similar to David’s suggestion.

I also left a comment about the documentation. And it probably needs a test case as well.

[gpshead]

TL;DR - We really need reliable tests for the exact behavior we want before coming up with patches.

attached is a patch (-gps01) that would do the same thing as Lib/smtpd.py does... But I'm not convinced it is a good idea.

Would forcing a socket.getaddrinfo() call from the constructor cause problems? This would be new behavior over what TCPServer did in the past. Could it trigger a blocking reverse DNS lookup where there wasn't one in the past?

What about when server_address[0] is ''? getaddrinfo() fails on that, but the existing code works and binds to 0.0.0.0. Presumably this should bind via AF_INET6 if the host supports it but a simple getaddrinfo() call doesn't tell us that.

[vadmium]

I also have reservations about using getaddrinfo() like this. Some potential problems:

1. IPv4-only compatibility: On my Linux computer, getaddrinfo("localhost", 0) returns AF_INET6 before AF_INET. Some programs, like Firefox and BSD netcat, will try both v4 and v6 and probably succeed (netcat first warned about “connection refused” for IPv4). However, other programs only support IPv4 at all or by default. Socat supports v6 but requires an explicit TCP6 or pf=ip6 argument. Gnu netcat only seems to support IPv4. This could also have been a problem with smtpd, or maybe SMTP clients are supposed to be smarter and it is okay (I am not familiar with the protocol).

2. Similarly, maybe getaddrinfo() could return AF_INET first even though the user made a subclass with address_family = AF_INET6.

3. It seems a waste to do a DNS lookup just to choose the address_family and throw away the resolved addresses, only to then call bind() which will do the lookup all over again. If DNS times out, the delay until error reporting will be twice as long.

4. getaddrinfo(("", None)) has a short delay (DNS timeout?) for me. The Python documentation says the empty string "" is special-cased to mean INADDR_ANY (IPv4). It also says there is no special case for the IPv6 equivalent (in6addr_any), although it does seem to work in some cases, including bind(). But getaddrinfo() would parse the string in the underlying platform, not in Python.

Some ideas:

1. Don’t look up hostnames, and only determine IPv6 if a numerical IP address is specified. I think this closer to the original proposal. Maybe use AI_NUMERICHOST? Or the simplistic test proposed in bpo-24209?

2. For the empty string address "", if the platform supports dual stack, use AF_INET6, bind to “::” and clear the IPV6_V6ONLY option. See bpo-3213.

3. Bind sockets to all addresses returned by getaddrinfo(), not just the first. But this would be a much bigger and more radical change. Maybe just something to dream about. :)

4. Add an address_family parameter to the constructor, so the user can change to AF_INET6 without subclassing.

[PaulMarks]

First off, the server_address=('localhost', port) case: this feature is fundamentally broken without support for multiple sockets, because the server can listen on at most one address, and any single choice will often be inconsistent with clients' expectations.

For the (ip, port) case, the only question is whether an IPv4 address should create an AF_INET or dualstack (IPV6_V6ONLY=0) AF_INET6 socket; the latter needs a.b.c.d -> ::ffff:a.b.c.d translation.

Then there's the typical ('', port) case. This should ideally create a server that accepts a connection on any supported family:

• If the system supports dualstack AF_INET6 sockets, use one.
• If the system only supports AF_INET sockets, use AF_INET.
• If the system only supports AF_INET6 sockets, use AF_INET6.
• If the system supports AF_INET and AF_INET6, but not dualstack, then you need multiple sockets. Or go with AF_INET and hope nobody needs IPv6 on such a system.

(The legacy-free simple approach is to hard-code AF_INET6 and let the OS decide whether the socket should be dualstack or IPv6-only, but that only supports 2/4 of the above cases.)

Users of "conn, addr = self.get_request()" often expect addr to be an IPv4 2-tuple, especially when handling IPv4 requests. So you have to normalize ('::ffff:192.0.2.1', 1234, 0, 0) to ('192.0.2.1', 1234). For real IPv6 requests, it's often more convenient to keep returning 2-tuples ('2001:db8::1', 1234) instead of 4-tuples.

Another thing to watch out for is this HTTPServer code:
host, port = self.socket.getsockname()[:2]
self.server_name = socket.getfqdn(host)

getfqdn() has a special case for '0.0.0.0', which doesn't recognize the IPv6 equivalents:
    >>> import socket
    >>> socket.getfqdn('0.0.0.0')
   'redacted.corp.google.com'
    >>> socket.getfqdn('::')
   '::'
   >>> socket.getfqdn('::ffff:0.0.0.0')
   '::ffff:0.0.0.0'

[vadmium]

Paul, if the user specifically wants to bind to a numeric IPv4 address, is there any advantage of choosing the dual-stack IPV6_V6ONLY=0 mode with a mapped ::ffff:a.b.c.d address?

[PaulMarks]

if the user specifically wants to bind to a numeric IPv4 address, is there any advantage of choosing the dual-stack [...]?

If you're in a position to write AF_INET6-only code, then dualstack sockets can make things a bit cleaner (one family for all IP communication). But given that Python couldn't reasonably drop support for AF_INET-only systems, there's not a compelling reason to prefer dualstack sockets for IPv4 stuff.

They're just two windows into the same kernel code, so the decision is mostly arbitrary.

However, Python likes to expose IP addresses as plain strings without transparent ::ffff:0.0.0.0/96 handling, which tends to make dualstack sockets a leaky abstraction. Ideally, you'd be able to talk to the kernel using AF_INET or AF_INET6 without normal users knowing the difference.

[nirs]

Doesn't it affect also 2.7, 3.6, 3.7, and 3.8?