Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bpo-37758: Extend unicodedata checksum tests to cover all of Unicode. #15125

Merged
merged 1 commit into from
Sep 12, 2019
Merged

bpo-37758: Extend unicodedata checksum tests to cover all of Unicode. #15125

merged 1 commit into from
Sep 12, 2019

Conversation

gnprice
Copy link
Contributor

@gnprice gnprice commented Aug 5, 2019 •
edited by bedevere-bot
Loading

Unicode has grown since Python first gained support for it,
when Unicode itself was still rather new.

This pair of test cases was added in commit 6a20ee7 back in 2000,
and they haven't needed to change much since then. But do change
them to look beyond the Basic Multilingual Plane (range(0x10000))
and cover all 17 planes of Unicode's final form.

This adds about 5 seconds to the test suite's runtime. Mark the
tests as CPU-using accordingly.

https://bugs.python.org/issue37758

@bedevere-bot bedevere-bot added tests Tests in the Lib/test dir awaiting review labels Aug 5, 2019
@benjaminp
Copy link
Contributor

This seems okay, though it now has conflicts with master.

@gnprice
bpo-37758: Extend unicodedata checksum tests to cover all of Unicode.
517484c 
Unicode has grown since Python first gained support for it,
when Unicode itself was still rather new.

This pair of test cases was added in commit 6a20ee7 back in 2000,
and they haven't needed to change much since then.  But do change
them to look beyond the Basic Multilingual Plane (range(0x10000))
and cover all 17 planes of Unicode's final form.

This adds about 5 seconds to the test suite's runtime.  Mark the
tests as CPU-using accordingly.
@gnprice gnprice force-pushed the pr-unicodedata-extend-tests branch from 3e06273 to 517484c Compare September 12, 2019 03:42
@gnprice
Copy link
Contributor Author

gnprice commented Sep 12, 2019

Thanks for the review -- rebased!

@benjaminp benjaminp merged commit 6954be8 into python:master Sep 12, 2019
DinoV pushed a commit to DinoV/cpython that referenced this pull request Sep 12, 2019
@gnprice @DinoV
closes bpo-37758: Extend unicodedata checksum tests to cover all of U…
1476fcb 
…nicode. (pythonGH-15125)

Unicode has grown since Python first gained support for it,
when Unicode itself was still rather new.

This pair of test cases was added in commit 6a20ee7 back in 2000,
and they haven't needed to change much since then.  But do change
them to look beyond the Basic Multilingual Plane (range(0x10000))
and cover all 17 planes of Unicode's final form.

This adds about 5 seconds to the test suite's runtime.  Mark the
tests as CPU-using accordingly.
@gnprice gnprice deleted the pr-unicodedata-extend-tests branch September 18, 2019 07:41
anthrotype added a commit to anthrotype/unicodedata2 that referenced this pull request Mar 17, 2020
@anthrotype
Update checksum test for Unicode 13; extend test to all of Unicode
d363217 
This commit combines the following two upstream patches:

python/cpython#18913
commit c77aa2d60b420747886f4258cf159bdbb7354100
Author: Benjamin Peterson <benjamin@python.org>
Date:   Tue Mar 10 21:18:33 2020 -0700

    bpo-39926: Update unicodedata checksum tests for Unicode 13.0 update. (GH-18913)

    I forget these tests required the cpu resource.

python/cpython#15125
commit 6954be815a16fad11d1d66be576865bbbeb2b97d
Author: Greg Price <gnprice@gmail.com>
Date:   Thu Sep 12 02:25:25 2019 -0700

    closes bpo-37758: Extend unicodedata checksum tests to cover all of Unicode. (GH-15125)

    Unicode has grown since Python first gained support for it,
    when Unicode itself was still rather new.

    This pair of test cases was added in commit 6a20ee7de back in 2000,
    and they haven't needed to change much since then.  But do change
    them to look beyond the Basic Multilingual Plane (range(0x10000))
    and cover all 17 planes of Unicode's final form.

    This adds about 5 seconds to the test suite's runtime.  Mark the
    tests as CPU-using accordingly.
anthrotype added a commit to fonttools/unicodedata2 that referenced this pull request Mar 19, 2020
@anthrotype
Update to Unicode 13.0; build wheels for Python 3.8 (#35)
0339410 
* Clean up and reduce visual clutter in the makeunicodedata scripts

python/cpython#7558

commit faa2948654d15a859bc4317e00730ff213295764
Author: Stefan Behnel <stefan_ml@behnel.de>
Date:   Sat Jun 1 21:49:03 2019 +0200

    Clean up and reduce visual clutter in the makeunicode.py script. (GH-7558)

* bpo-37760: Factor out the basic UCD parsing logic of makeunicodedata. (GH-15130)

python/cpython#15130
commit ef2af1ad44be0542a47270d5173a0b920c3a450d
Author: Greg Price <gnprice@gmail.com>
Date:   Mon Aug 12 22:20:56 2019 -0700

    bpo-37760: Factor out the basic UCD parsing logic of makeunicodedata. (GH-15130)

    There were 10 copies of this, and almost as many distinct versions of
    exactly how it was written.  They're all implementing the same
    standard.  Pull them out to the top, so the more interesting logic
    that remains becomes easier to read.

~~~

I removed the type hints from UcdFile class to apply the same patch to both python 2 and 3

* bpo-37760: Constant-fold some old options in makeunicodedata. (GH-15129)

python/cpython#15129
commit 99d208efed97e02d813e8166925b998bbd0d3993 (HEAD)
Author: Greg Price <gnprice@gmail.com>
Date:   Mon Aug 12 22:59:30 2019 -0700

    bpo-37760: Constant-fold some old options in makeunicodedata. (GH-15129)

    The `expand` option was introduced in 2000 in commit fad27aee1.
    It appears to have been always set since it was committed, and
    what it does is tell the code to do something essential.  So,
    just always do that, and cut the option.

    Also cut the `linebreakprops` option, which isn't consulted anymore.

* bpo-37760: Factor out standard range-expanding logic in makeunicodedata. (GH-15248)

python/cpython#15248
commit c03e698c344dfc557555b6b07a3ee2702e45f6ee (HEAD)
Author: Greg Price <gnprice@gmail.com>
Date:   Tue Aug 13 19:28:38 2019 -0700

    bpo-37760: Factor out standard range-expanding logic in makeunicodedata. (GH-15248)

    Much like the lower-level logic in commit ef2af1ad4, we had
    4 copies of this logic, written in a couple of different ways.
    They're all implementing the same standard, so write it just once.

* bpo-37760: Avoid cluttering work tree with downloaded Unicode files. (GH-15128)

python/cpython#15128
commit 3e4498d35c34aeaf4a9c3d57509b0d3277048ac6
Author: Greg Price <gnprice@gmail.com>
Date:   Wed Aug 14 18:18:53 2019 -0700

    bpo-37760: Avoid cluttering work tree with downloaded Unicode files. (GH-15128)

* Convert from length-18 lists to namedtuple, in makeunicodedata. (GH-15265)

Adapted from: python/cpython#15265

commit a65678c5c90002c5e40fa82746de07e6217df625
Author: Greg Price <gnprice@gmail.com>
Date:   Thu Sep 12 02:23:43 2019 -0700

    bpo-37760: Convert from length-18 lists to a dataclass, in makeunicodedata. (GH-15265)

    Now the fields have names!  Much easier to keep straight as a
    reader than the elements of an 18-tuple.

    Runs about 10-15% slower: from 10.8s to 12.3s, on my laptop.
    Fortunately that's perfectly fine for this maintenance script.

~~~

The original patch uses dataclasses but I use namedtuple here so that it
works on both python 2 and 3.

* closes bpo-39926: Update Unicode to 13.0.0. (GH-18910)

Fixes #34

Adapted from: python/cpython#18910
commit 051b9d08d1e6a8b1022a2bd9166be51c0b152698
Author: Benjamin Peterson <benjamin@python.org>
Date:   Tue Mar 10 20:41:34 2020 -0700

    closes bpo-39926: Update Unicode to 13.0.0. (GH-18910)

* Update some www.unicode.org URLs to use HTTPS. (GH-18912)

Adapted from: python/cpython#18912
commit 51796e5d2632e6ada81ca677b4153f4ccd490702
Author: Benjamin Peterson <benjamin@python.org>
Date:   Tue Mar 10 21:10:59 2020 -0700

    Update some www.unicode.org URLs to use HTTPS. (GH-18912)

* Update checksum test for Unicode 13; extend test to all of Unicode

This commit combines the following two upstream patches:

python/cpython#18913
commit c77aa2d60b420747886f4258cf159bdbb7354100
Author: Benjamin Peterson <benjamin@python.org>
Date:   Tue Mar 10 21:18:33 2020 -0700

    bpo-39926: Update unicodedata checksum tests for Unicode 13.0 update. (GH-18913)

    I forget these tests required the cpu resource.

python/cpython#15125
commit 6954be815a16fad11d1d66be576865bbbeb2b97d
Author: Greg Price <gnprice@gmail.com>
Date:   Thu Sep 12 02:25:25 2019 -0700

    closes bpo-37758: Extend unicodedata checksum tests to cover all of Unicode. (GH-15125)

    Unicode has grown since Python first gained support for it,
    when Unicode itself was still rather new.

    This pair of test cases was added in commit 6a20ee7de back in 2000,
    and they haven't needed to change much since then.  But do change
    them to look beyond the Basic Multilingual Plane (range(0x10000))
    and cover all 17 planes of Unicode's final form.

    This adds about 5 seconds to the test suite's runtime.  Mark the
    tests as CPU-using accordingly.

* test_unicodedata2: add unichr for 'narrow' python builds

* Update multibuild to latest 'devel' branch

* Build and run tests on Python 3.8

* .travis.yml: remove implicit job

or else is rejected with "Build config did not create any jobs"

travis-ci/travis-ci#8536

* test_unicodedata2: do not import test.support.requires_resource

import fails for some reason on some older 2.7 versions, see
https://travis-ci.org/github/mikekap/unicodedata2/jobs/663493029

It should not make any difference without this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@gnprice @benjaminp @the-knights-who-say-ni @bedevere-bot