Skip to content

bpo-34788: Add support for scoped IPv6 addresses #13772

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 61 commits into from
Feb 26, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
eb4a407
bpo-34788 Add tests for ipv6 scope_id support
May 31, 2019
8a5c94f
bpo-34788 Add support for scoped IPv6 addresses
May 31, 2019
a841e9a
bpo-34788 Add documentation for scope id support
May 31, 2019
9a3f7ec
bpo-34788 Refactor address split and validation
May 31, 2019
a17538a
bpo-34788 Convert scope_id to property
Jun 3, 2019
6da269e
bpo-34788 Update documentation
Jun 3, 2019
35750de
bpo-34788 Remove redundant check
Jun 3, 2019
681cfa5
bpo-34788 Fix newline
Jun 3, 2019
fb99b6b
bpo-34788 Documentation fix
Jun 6, 2019
d8bb674
📜🤖 Added by blurb_it.
blurb-it[bot] Jul 17, 2019
ccd83ed
bpo-34788 Update NEWS
opavlyuk Jul 17, 2019
dc540e9
bpo-34788 Fix NEWS filename
opavlyuk Jul 17, 2019
c03d4a5
bpo-34788 Add sample IPv6 to ignore list
Jul 17, 2019
c12e533
bpo-34788 Add sample IPv6 to ignore list
Jul 17, 2019
033f031
bpo-34788 Update Misc/NEWS.d
opavlyuk Jul 18, 2019
27c5cf4
bpo-34788 Refactor condition
opavlyuk Jul 18, 2019
3b495c0
bpo-34788 Fix excessive property call
opavlyuk Jul 18, 2019
1de1ae6
Merge branch 'master' into fix-issue-34788
opavlyuk Aug 9, 2019
b0a4fe9
bpo-34788 Mention RFC 4007 in the docstring
opavlyuk Aug 12, 2019
ac19fad
bpo-34788 Remove unnecessary comma
opavlyuk Aug 12, 2019
25d3ac7
bpo-34788 Add scope_id check
opavlyuk Aug 12, 2019
66db707
bpo-34788 Add tests for invalid scope id
opavlyuk Aug 12, 2019
064e019
bpo-34788 Improve scoped IPv6 samples
Aug 14, 2019
ef91d9c
bpo-34788 Remove trailing whitespaces
Aug 14, 2019
d0eb91b
bpo-34788 Improve scope_id documentation
Aug 14, 2019
0735d3f
bpo-34788 Fix susp-ignored
Aug 14, 2019
fb47e34
bpo-34788 Improve IPv6Address documentation
Aug 14, 2019
b8dffe5
bpo-34788 Move self._scope_id initialization under if statements
Sep 16, 2019
fc2425f
bpo-34788 Remove newline
Sep 16, 2019
f9d38c0
bpo-34788 Improve documentation
Sep 16, 2019
d8972af
bpo-34788 Consider scope id while comparing IPv6 addresses
Nov 26, 2019
0b37c30
bpo-34788 Update susp-ignored.csv
Nov 26, 2019
41ce543
bpo-34788 Update documentation
Nov 26, 2019
7f3fae8
bpo-34788 Update susp-ignored
Nov 26, 2019
6c5a845
bpo-34788 Update susp-ignored
Nov 26, 2019
e569cd5
bpo-34788 Update susp-ignored
Nov 26, 2019
3982d78
bpo-34788 Update susp-ignored
Nov 26, 2019
6b5198f
bpo-34788 Update susp-ignored
Nov 26, 2019
97a7707
bpo-34788 Update tests according to new comparison policy
Nov 27, 2019
6339732
bpo-34788 Update documentation
Nov 27, 2019
eff04f6
bpo-34788 Add 3.9 whatsnew record
Nov 27, 2019
06e66e9
Merge branch 'master' into fix-issue-34788
opavlyuk Nov 28, 2019
c25a327
bpo-34788 Improve __hash__()
Nov 28, 2019
84f4b3d
Merge branch 'fix-issue-34788' of github.com:opavlyuk/cpython into fi…
Nov 28, 2019
c6695bf
bpo-34788 Update whatsnew
Nov 28, 2019
d30ea41
Update Doc/whatsnew/3.9.rst
opavlyuk Nov 28, 2019
e971130
Update Doc/whatsnew/3.9.rst
opavlyuk Nov 28, 2019
22c310e
Update Doc/whatsnew/3.9.rst
opavlyuk Nov 29, 2019
5882d06
bpo-34788 Split conditional statement
opavlyuk Nov 29, 2019
2c6fd54
bpo-34788 Fix eq
Nov 29, 2019
0ec1572
bpo-34788 Unify docstring styles
Nov 29, 2019
5530a79
bpo-34788 Remove unnecessary arg
Nov 29, 2019
2f7de77
bpo-34788 Improve comparison
Dec 2, 2019
ef42a2d
bpo-34788 Fix typo
Dec 2, 2019
641bc6b
bpo-34788 Unify scope id naming
Dec 3, 2019
34728d2
bpo-34788 Unify naming
Dec 3, 2019
af90771
Update Doc/whatsnew/3.9.rst
opavlyuk Dec 11, 2019
e041c2e
Merge branch 'master' into fix-issue-34788
opavlyuk Jan 14, 2020
0e250bf
Merge remote-tracking branch 'origin/master' into fix-issue-34788
Feb 14, 2020
2c4d3d5
Cleanup Whatsnew
opavlyuk Feb 26, 2020
555f59f
Fix typo
opavlyuk Feb 26, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 24 additions & 2 deletions Doc/library/ipaddress.rst
Original file line number Diff line number Diff line change
Expand Up @@ -217,11 +217,20 @@ write code that handles both IP versions correctly. Address objects are
:RFC:`4291` for details. For example,
``"0000:0000:0000:0000:0000:0abc:0007:0def"`` can be compressed to
``"::abc:7:def"``.

Optionally, the string may also have a scope zone ID, expressed
with a suffix ``%scope_id``. If present, the scope ID must be non-empty,
and may not contain ``%``.
See :RFC:`4007` for details.
For example, ``fe80::1234%1`` might identify address ``fe80::1234`` on the first link of the node.
2. An integer that fits into 128 bits.
3. An integer packed into a :class:`bytes` object of length 16, big-endian.


>>> ipaddress.IPv6Address('2001:db8::1000')
IPv6Address('2001:db8::1000')
>>> ipaddress.IPv6Address('ff02::5678%1')
IPv6Address('ff02::5678%1')

.. attribute:: compressed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this documentation on compressed is no longer accurate. Is that right? If so, let's update it.

(Noticed this while skimming the doc just now to answer the question about where to make this feature visible with examples.)

Expand Down Expand Up @@ -268,6 +277,12 @@ write code that handles both IP versions correctly. Address objects are
``::FFFF/96``), this property will report the embedded IPv4 address.
For any other address, this property will be ``None``.

.. attribute:: scope_id

For scoped addresses as defined by :RFC:`4007`, this property identifies
the particular zone of the address's scope that the address belongs to,
as a string. When no scope zone is specified, this property will be ``None``.

.. attribute:: sixtofour

For addresses that appear to be 6to4 addresses (starting with
Expand Down Expand Up @@ -299,6 +314,8 @@ the :func:`str` and :func:`int` builtin functions::
>>> int(ipaddress.IPv6Address('::1'))
1

Note that IPv6 scoped addresses are converted to integers without scope zone ID.


Operators
^^^^^^^^^
Expand All @@ -311,15 +328,20 @@ IPv6).
Comparison operators
""""""""""""""""""""

Address objects can be compared with the usual set of comparison operators. Some
examples::
Address objects can be compared with the usual set of comparison operators.
Same IPv6 addresses with different scope zone IDs are not equal.
Some examples::

>>> IPv4Address('127.0.0.2') > IPv4Address('127.0.0.1')
True
>>> IPv4Address('127.0.0.2') == IPv4Address('127.0.0.1')
False
>>> IPv4Address('127.0.0.2') != IPv4Address('127.0.0.1')
True
>>> IPv6Address('fe80::1234') == IPv6Address('fe80::1234%1')
False
>>> IPv6Address('fe80::1234%1') != IPv6Address('fe80::1234%2')
True


Arithmetic operators
Expand Down
20 changes: 10 additions & 10 deletions Doc/library/socket.rst
Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,15 @@ created. Socket addresses are represented as follows:
Python programs.

- For :const:`AF_INET6` address family, a four-tuple ``(host, port, flowinfo,
scopeid)`` is used, where *flowinfo* and *scopeid* represent the ``sin6_flowinfo``
scope_id)`` is used, where *flowinfo* and *scope_id* represent the ``sin6_flowinfo``
and ``sin6_scope_id`` members in :const:`struct sockaddr_in6` in C. For
:mod:`socket` module methods, *flowinfo* and *scopeid* can be omitted just for
backward compatibility. Note, however, omission of *scopeid* can cause problems
:mod:`socket` module methods, *flowinfo* and *scope_id* can be omitted just for
backward compatibility. Note, however, omission of *scope_id* can cause problems
in manipulating scoped IPv6 addresses.

.. versionchanged:: 3.7
For multicast addresses (with *scopeid* meaningful) *address* may not contain
``%scope`` (or ``zone id``) part. This information is superfluous and may
For multicast addresses (with *scope_id* meaningful) *address* may not contain
``%scope_id`` (or ``zone id``) part. This information is superfluous and may
be safely omitted (recommended).

- :const:`AF_NETLINK` sockets are represented as pairs ``(pid, groups)``.
Expand Down Expand Up @@ -738,7 +738,7 @@ The :mod:`socket` module also offers various network-related services:
:const:`AI_CANONNAME` is part of the *flags* argument; else *canonname*
will be empty. *sockaddr* is a tuple describing a socket address, whose
format depends on the returned *family* (a ``(address, port)`` 2-tuple for
:const:`AF_INET`, a ``(address, port, flow info, scope id)`` 4-tuple for
:const:`AF_INET`, a ``(address, port, flowinfo, scope_id)`` 4-tuple for
:const:`AF_INET6`), and is meant to be passed to the :meth:`socket.connect`
method.

Expand All @@ -759,7 +759,7 @@ The :mod:`socket` module also offers various network-related services:

.. versionchanged:: 3.7
for IPv6 multicast addresses, string representing an address will not
contain ``%scope`` part.
contain ``%scope_id`` part.

.. function:: getfqdn([name])

Expand Down Expand Up @@ -827,8 +827,8 @@ The :mod:`socket` module also offers various network-related services:
or numeric address representation in *host*. Similarly, *port* can contain a
string port name or a numeric port number.

For IPv6 addresses, ``%scope`` is appended to the host part if *sockaddr*
contains meaningful *scopeid*. Usually this happens for multicast addresses.
For IPv6 addresses, ``%scope_id`` is appended to the host part if *sockaddr*
contains meaningful *scope_id*. Usually this happens for multicast addresses.

For more information about *flags* you can consult :manpage:`getnameinfo(3)`.

Expand Down Expand Up @@ -1354,7 +1354,7 @@ to sockets.

.. versionchanged:: 3.7
For multicast IPv6 address, first item of *address* does not contain
``%scope`` part anymore. In order to get full IPv6 address use
``%scope_id`` part anymore. In order to get full IPv6 address use
:func:`getnameinfo`.

.. method:: socket.recvmsg(bufsize[, ancbufsize[, flags]])
Expand Down
2 changes: 2 additions & 0 deletions Doc/tools/susp-ignored.csv
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,8 @@ library/ipaddress,,:db8,>>> ipaddress.IPv6Address('2001:db8::1000')
library/ipaddress,,::,>>> ipaddress.IPv6Address('2001:db8::1000')
library/ipaddress,,:db8,IPv6Address('2001:db8::1000')
library/ipaddress,,::,IPv6Address('2001:db8::1000')
library/ipaddress,,::,IPv6Address('ff02::5678%1')
library/ipaddress,,::,fe80::1234
library/ipaddress,,:db8,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer"
library/ipaddress,,::,">>> ipaddress.ip_address(""2001:db8::1"").reverse_pointer"
library/ipaddress,,::,"""::abc:7:def"""
Expand Down
9 changes: 9 additions & 0 deletions Doc/whatsnew/3.9.rst
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,15 @@ now raises :exc:`ImportError` instead of :exc:`ValueError` for invalid relative
import attempts.
(Contributed by Ngalim Siregar in :issue:`37444`.)

ipaddress
---------

:mod:`ipaddress` now supports IPv6 Scoped Addresses (IPv6 address with suffix ``%<scope_id>``).

Scoped IPv6 addresses can be parsed using :class:`ipaddress.IPv6Address`.
If present, scope zone ID is available through the :attr:`~ipaddress.IPv6Address.scope_id` attribute.
(Contributed by Oleksandr Pavliuk in :issue:`34788`.)

math
----

Expand Down
55 changes: 53 additions & 2 deletions Lib/ipaddress.py
Original file line number Diff line number Diff line change
Expand Up @@ -1836,6 +1836,26 @@ def _reverse_pointer(self):
reverse_chars = self.exploded[::-1].replace(':', '')
return '.'.join(reverse_chars) + '.ip6.arpa'

@staticmethod
def _split_scope_id(ip_str):
"""Helper function to parse IPv6 string address with scope id.

See RFC 4007 for details.

Args:
ip_str: A string, the IPv6 address.

Returns:
(addr, scope_id) tuple.

"""
addr, sep, scope_id = ip_str.partition('%')
if not sep:
scope_id = None
elif not scope_id or '%' in scope_id:
raise AddressValueError('Invalid IPv6 address: "%r"' % ip_str)
return addr, scope_id

@property
def max_prefixlen(self):
return self._max_prefixlen
Expand All @@ -1849,7 +1869,7 @@ class IPv6Address(_BaseV6, _BaseAddress):

"""Represent and manipulate single IPv6 Addresses."""

__slots__ = ('_ip', '__weakref__')
__slots__ = ('_ip', '_scope_id', '__weakref__')

def __init__(self, address):
"""Instantiate a new IPv6 address object.
Expand All @@ -1872,21 +1892,52 @@ def __init__(self, address):
if isinstance(address, int):
self._check_int_address(address)
self._ip = address
self._scope_id = None
return

# Constructing from a packed address
if isinstance(address, bytes):
self._check_packed_address(address, 16)
self._ip = int.from_bytes(address, 'big')
self._scope_id = None
return

# Assume input argument to be string or any object representation
# which converts into a formatted IP string.
addr_str = str(address)
if '/' in addr_str:
raise AddressValueError("Unexpected '/' in %r" % address)
addr_str, self._scope_id = self._split_scope_id(addr_str)

self._ip = self._ip_int_from_string(addr_str)

def __str__(self):
ip_str = super().__str__()
return ip_str + '%' + self._scope_id if self._scope_id else ip_str
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this adds the scoped syntax, with %, to the result of str on an address.

Are there places where that's likely to wind up in a URL? The RFC in section 11.7 expressed concern about this syntax's interaction with URLs -- in particular, the % will probably need to be percent-encoded.

I think if that causes a problem, it's a bug in the code that made the URL; so we can't do everything to prevent it. But:

  • To help give notice to third-party code, we should be sure to make the scoped-address case visible in the docs and in What's New.
  • We should check the stdlib itself for such bugs. 🙂

... OK, it looks like nothing else in the stdlib refers to this module, so that takes care of point 2.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you clarify you suggestion, please?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure -- the request is this bit:

To help give notice to third-party code, we should be sure to make the scoped-address case visible in the docs and in What's New.

Concretely that means something like:

  • This new feature should be mentioned in the "What's New in 3.9" document. That's Doc/whatsnew/3.9.rst. (After all, it's one of the nice new things we'll have in 3.9!)
  • Each of the main places in the docs that have a series of examples should include an example with a scoped address, so that it's visible that these addresses with % in them are a thing that can happen. That means something like:
    • The initial "Convenience factory functions" section in the module doc. For a lot of people this may be the one section they really read. One example in either the ip_address or the ip_network case should be enough.
    • The docs bit on the IPv6Address constructor. Already there in this version -- thanks!
    • The "Conversion to Strings and Integers" section in the doc.
    • The first block of examples in the "IP Host Addresses" section at the top of Doc/howto/ipaddress.rst aka here. That's prominently linked to from the module doc, so a lot of people may rely more on that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@gnprice , great thanks, really very helpful. In progress.


def __hash__(self):
return hash((self._ip, self._scope_id))

def __eq__(self, other):
address_equal = super().__eq__(other)
if address_equal is NotImplemented:
return NotImplemented
if not address_equal:
return False
return self._scope_id == getattr(other, '_scope_id', None)

@property
def scope_id(self):
"""Identifier of a particular zone of the address's scope.

See RFC 4007 for details.

Returns:
A string identifying the zone of the address if specified, else None.

"""
return self._scope_id

@property
def packed(self):
"""The binary representation of this address."""
Expand Down Expand Up @@ -2040,7 +2091,7 @@ def hostmask(self):
return self.network.hostmask

def __str__(self):
return '%s/%d' % (self._string_from_ip_int(self._ip),
return '%s/%d' % (super().__str__(),
self._prefixlen)

def __eq__(self, other):
Expand Down
Loading