str.isspace should use the Unicode White_Space property

[abalkin]

BPO	18236
Nosy	@malemburg, @loewis, @terryjreedy, @abalkin, @vstinner, @ezio-melotti, @bitdancer, @vadmium, @serhiy-storchaka, @animalize, @gnprice
PRs	bpo-18236: Adjust str.isspace to use Unicode's White_Space property. #16254
Files	42973dfea391.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-06-17.01:06:30.764>
labels = ['interpreter-core', 'type-bug', '3.9', 'expert-unicode']
title = 'str.isspace should use the Unicode White_Space property'
updated_at = <Date 2019-09-18.06:02:16.653>
user = 'https://github.com/abalkin'

bugs.python.org fields:

activity = <Date 2019-09-18.06:02:16.653>
actor = 'Greg Price'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core', 'Unicode']
creation = <Date 2013-06-17.01:06:30.764>
creator = 'belopolsky'
dependencies = []
files = ['30680']
hgrepos = ['201']
issue_num = 18236
keywords = ['patch']
message_count = 18.0
messages = ['191303', '191612', '191647', '191648', '191649', '191650', '191652', '191687', '191689', '191706', '191739', '191746', '221919', '230183', '349213', '349214', '349233', '350414']
nosy_count = 11.0
nosy_names = ['lemburg', 'loewis', 'terry.reedy', 'belopolsky', 'vstinner', 'ezio.melotti', 'r.david.murray', 'martin.panter', 'serhiy.storchaka', 'malin', 'Greg Price']
pr_nums = ['16254']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue18236'
versions = ['Python 3.9']

[abalkin]

ASCII information separators, hex codes 1C through 1F are classified as space:

>>> all(c.isspace() for c in '\N{FS}\N{GS}\N{RS}\N{US}')
True

but int()/float() do not accept strings with leading or trailing separators:

>>> int('123\N{RS}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '123\x1e'

This is probably because corresponding bytes values are not classified as whitespace:

>>> any(c.encode().isspace() for c in '\N{FS}\N{GS}\N{RS}\N{US}')
False

[terryjreedy]

You stated facts: what is your proposal?

The fact that unicode calls characters 'space' does not make then whitespace as commonly understood, or as defined by C, or even as defined by the Unicode database. Unicode apparently has a WSpace property. According to the table in
https://en.wikipedia.org/wiki/Whitespace_%28computer_science%29
1C - 1F are not included by that definition either. For ascii chars, that table matches the C definition, with \r included.

So I think your implied proposal to treat them as whitespace (in strings but not bytes) should be rejected as invalid. For 3.x, the manual should specify that it follows the C definition of 'whitespace' (\r included) for bytes and the extended unicode definition for strings.

>>> int('3\r')
3
>>> int('3\u00a0')
3
>>> int('3\u2000')
3
>>> int(b'3\r')
3
>>> int(b'3\u00a0')
Traceback (most recent call last):
  File "<pyshell#10>", line 1, in <module>
    int(b'3\u00a0')
ValueError: invalid literal for int() with base 10: '3\\u00a0'

[abalkin]

You stated facts: what is your proposal?

There is a bug somewhere. We cannot simultaneously have

>>> '\N{RS}'.isspace()
True

and not accept '\N{RS}' as whitespace when parsing numbers.

I believe int(x) should be equivalent to int(x.strip()). This is not the case now:

>>> '123\N{RS}'.strip()
'123'
>>> int('123\N{RS}')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '123\x1e'

The reason I did not clearly state my proposal is because I am not sure whether bytes.isspace or str.isspace is correct, but I don't see any justification for having them defined differently in the ASCII range.

[terryjreedy]

I see your point now. Since RS is not whitespace by any definition I knew of previously, why is RS.isspace True?

Apparent answer: Doc says '''Return true if there are only whitespace characters in the string and there is at least one character, false otherwise. Whitespace characters are those characters defined in the Unicode character database as “Other” or “Separator” and those with bidirectional property being one of “WS”, “B”, or “S”.''' I suspect this is a more expansive definition than WSpace chars, which seems to be the one used by int(), but you could check the int code.

Bytes docs says: "Whenever a bytes or bytearray method needs to interpret the bytes as characters (e.g. the is...() methods, split(), strip()), the ASCII character set is assumed (text strings use Unicode semantics)."

This says to me that str.isxxx and bytes.isxxx should match on ascii chars and not otherwise. That would happen is the bytes methods check for all ascii and decoded to unicode and used str method. Since they do not match, bytes must do something different.

I think there is one definite bug: the discrepancy between str.isspace and bytes.isspace. There is possibly another bug: the discrepancy between 'whitespace' for str.isspace and int/float. After pinning down the details, I think you should ask how to resolve these on py-dev, and which versions to patch.

[abalkin]

It looks like str.isspace() is incorrect. The proper definition of unicode whitespace seems to include 26 characters:

# ================================================

0009..000D ; White_Space # Cc [5] <control-0009>..<control-000D>
0020 ; White_Space # Zs SPACE
0085 ; White_Space # Cc <control-0085>
00A0 ; White_Space # Zs NO-BREAK SPACE
1680 ; White_Space # Zs OGHAM SPACE MARK
180E ; White_Space # Zs MONGOLIAN VOWEL SEPARATOR
2000..200A ; White_Space # Zs [11] EN QUAD..HAIR SPACE
2028 ; White_Space # Zl LINE SEPARATOR
2029 ; White_Space # Zp PARAGRAPH SEPARATOR
202F ; White_Space # Zs NARROW NO-BREAK SPACE
205F ; White_Space # Zs MEDIUM MATHEMATICAL SPACE
3000 ; White_Space # Zs IDEOGRAPHIC SPACE

# Total code points: 26

http://www.unicode.org/Public/UNIDATA/PropList.txt

Python's str.isspace() uses the following definition: "Whitespace characters are those characters defined in the Unicode character database as “Other” or “Separator” and those with bidirectional property being one of “WS”, “B”, or “S”."

Information separators are swept in because they have bidirectional property "B":

>>> unicodedata.bidirectional('\N{RS}')
'B'

See also bpo-10587.

[abalkin]

I did a little more investigation and it looks like information separators have been included in whitespace since unicode type was first implemented in Python:

guido 11967 Fri Mar 10 22:52:46 2000 +0000: /* Returns 1 for Unicode characters having the type 'WS', 'B' or 'S',
guido 11967 Fri Mar 10 22:52:46 2000 +0000: 0 otherwise. */
guido 11967 Fri Mar 10 22:52:46 2000 +0000:
guido 11967 Fri Mar 10 22:52:46 2000 +0000: int _PyUnicode_IsWhitespace(register const Py_UNICODE ch)
guido 11967 Fri Mar 10 22:52:46 2000 +0000: {
guido 11967 Fri Mar 10 22:52:46 2000 +0000: switch (ch) {
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x0009: /* HORIZONTAL TABULATION */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x000A: /* LINE FEED */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x000B: /* VERTICAL TABULATION */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x000C: /* FORM FEED */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x000D: /* CARRIAGE RETURN */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x001C: /* FILE SEPARATOR */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x001D: /* GROUP SEPARATOR */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x001E: /* RECORD SEPARATOR */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x001F: /* UNIT SEPARATOR */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x0020: /* SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x1680: /* OGHAM SPACE MARK */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2000: /* EN QUAD */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2001: /* EM QUAD */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2002: /* EN SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2003: /* EM SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2004: /* THREE-PER-EM SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2005: /* FOUR-PER-EM SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2006: /* SIX-PER-EM SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2007: /* FIGURE SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2008: /* PUNCTUATION SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2009: /* THIN SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x200A: /* HAIR SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x2028: /* LINE SEPARATOR */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x202F: /* NARROW NO-BREAK SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: case 0x3000: /* IDEOGRAPHIC SPACE */
guido 11967 Fri Mar 10 22:52:46 2000 +0000: return 1;
guido 11967 Fri Mar 10 22:52:46 2000 +0000: default:
guido 11967 Fri Mar 10 22:52:46 2000 +0000: return 0;
guido 11967 Fri Mar 10 22:52:46 2000 +0000: }
guido 11967 Fri Mar 10 22:52:46 2000 +0000: }
guido 11967 Fri Mar 10 22:52:46 2000 +0000:

(hg blame -u -d -n -r 11967 Objects/unicodectype.c)

[abalkin]

Martin v. Löwis wrote at bpo-13391 (msg147634):

I do think that _PyUnicode_IsWhitespace should use the White_Space
property (from PropList.txt). I'm not quite sure how they computed
that property (or whether it's manually curated). Since that's a
behavioral change, it can only go into 3.3.

I am adding Martin and Ezio to the "nosy."

[loewis]

I stand by that comment: IsWhiteSpace should use the Unicode White_Space property. Since FS/GS/RS/US are not in the White_Space property, it's correct that the int conversion fails. It's incorrect that .isspace() gives true.

There are really several bugs here:

• .isspace doesn't use the White_List property
• int conversion ultimately uses Py_ISSPACE, which conceptually could deviate from the Unicode properties (as it is byte-based). This is not really an issue, since they indeed match.

I propose to fix this by parsing PropList.txt, and generating _PyUnicode_IsWhitespace based on the White_Space property. For efficiency, it should also generate a fast-lookup array for the ASCII case, or just use _Py_ctype_table (with a comment that this table needs to match PropList White_Space). _Py_ascii_whitespace should go.

Contributions are welcome.

[malemburg]

I agree with Martin.

At the time Unicode was added to Python, there was no single Unicode property for white space, so I had to deduce this from the other available properties.

Now that we have a white space property in Unicode, we should start using it. Fortunately, the difference in Python's set of white space chars and the ones having the Unicode white space property are minimal.

[abalkin]

I have updated the title to focus this issue on the behavior of str.isspace(). I'll pick up remaining int/float issues in bpo-10581.