gh-101372: Fix unicodedata.is_normalized to properly handle the UCD 3…

[corona10]

….2.0

• Issue: unicodedata: is_normalized claims nothing is normalized in any form when using the 3.2.0 database #101372

[corona10]

@serhiy-storchaka

All ranges of characters are candidates for testing.
So I decide to choose a sampling approach rather than choose specific cases.

Test script

import unicodedata

with open('foo.out', 'w') as f:
    for x in range(0x110000):
        for form in ('NFC', 'NFD', 'NFKC', 'NFKD'):
            norm = unicodedata.ucd_3_2_0.normalize(form, chr(x))
            if not unicodedata.ucd_3_2_0.is_normalized(form, norm):
                f.write(f'{str(x)},{form}\n')

AS-IS

(.oss) ➜  cpython git:(main) ✗ ./python.exe gh-101372.py
(.oss) ➜  cpython git:(main) ✗ wc -l foo.out
 4456448 foo.out

TO-BE

(.oss) ➜  cpython git:(gh-101372) ✗ ./python.exe gh-101372.py
(.oss) ➜  cpython git:(gh-101372) ✗ wc -l foo.out
       0 foo.out

[corona10]

@serhiy-storchaka I will merge this PR by next week, please let me know if there need some changes

[serhiy-storchaka]

I am not happy with provided tests.

Testing all range of Unicode characters is slow (few seconds on my computer), it should be decorated with @requires_resource('cpu') if performed. Testing only small random sample can miss errors, and the test result will be hardly reproducible. It works for this issue, because is_normalized() was broken for most of characters, but it could not work for other types of bugs.

The test for multicharacter string is not what I meant. It should not only test all normalized sequences, but also non-normalized sequences. For example, '\ufb2c' is normalized to '\u05e9\u05bc\u05c1'. Therefore, '\ufb2c' should be not normalized, and '\u05e9\u05bc\u05c1' should be normalized. But '\u05e9\u05c1\u05bc', created by swapping the last two characters, is normalized to the same sequence '\u05e9\u05bc\u05c1', therefore it should be not normalized, besides it looks exactly the same as the original character. I think we need such kind of tests.

I tried to write more interesting tests for is_normalized(), and have found that the UCD 3.2.0 is mostly not tested. Also, there are not many tests for differences between UCD 3.2.0 and the current version. I am writing new tests.

I propose to merge your PR without tests. The bugfix itself is obvious, and the tests I will add later.

[corona10]

I propose to merge your PR without tests. The bugfix itself is obvious, and the tests I will add later.

Okay got it, Please let me know once you submit the patch for test codes. I may learn a lot from the patch.

[miss-islington]

Thanks @corona10 for the PR 🌮🎉.. I'm working now to backport this PR to: 3.10, 3.11.
🐍🍒⛏🤖

[bedevere-bot]

GH-101597 is a backport of this pull request to the 3.11 branch.

[bedevere-bot]

GH-101598 is a backport of this pull request to the 3.10 branch.