bpo-41391: Make test_unicodedata pass when running without network

[yan12125]

The self.fail() statement below was added to make HTTP 404 error visible
[1]. After this patch HTTP 404 still fails the test as
transient_internet() does not filter 4xx errors.

[1] https://bugs.python.org/issue29887

A sample failure log before patch:

0:05:28 load avg: 1.21 [376/423/11] test_unicodedata failed
test test_unicodedata failed -- Traceback (most recent call last):
  File "/data/local/tmp/lib/python3.10/urllib/request.py", line 1342, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
socket.gaierror: [Errno 7] No address associated with hostname

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/lib/python3.10/test/test_unicodedata.py", line 329, in test_normalization
    testdata = open_urlresource(TESTDATAURL, encoding="utf-8",
urllib.error.URLError: <urlopen error [Errno 7] No address associated with hostname>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/local/tmp/lib/python3.10/test/test_unicodedata.py", line 335, in test_normalization
    self.fail(f"Could not retrieve {TESTDATAURL}")
AssertionError: Could not retrieve http://www.pythontest.net/unicode/13.0.0/NormalizationTest.txt

The log is grabbed from an Android emulator running Android 10 x86_64
and testing CPython commit af08db7.

https://bugs.python.org/issue41391

[yan12125]

I updated the commit message to (hopefully) make the intention clearer by mentioning a related issue https://bugs.python.org/issue29887.

[terryjreedy]

Someone needs to check one use of support.socket_helper.transient_internet

[vstinner]

open_urlresource() skips the test if the urlfetch resource is disallowed. By default, all resources are enabled.

Maybe the default should exclude urlfetch? Or the "network" resource should be reused?

https://docs.python.org/dev/library/test.html#test.support.requires

[vstinner]

I'm not sure that the code is wrong. Maybe the error message should explain how to disable the test resource "urlfetch" instead?

cc @pablogsal @zware

[zware]

Hmm, I think transient_internet actually does exactly what we want here; it will skip the test if the network connection fails (or if the server fails, returning a 5xx HTTP error), but should raise the original HTTPError if the requested file is missing (404).

This change LGTM, but I have not actually tested it to confirm my reading of the code.

[yan12125]

Oops, I found that I added description about HTTP 4xx errors to the new commit message but forgot to also update the pull request description. That's done now.

I've also checked that 404 errors from the server fails the test. With the following changes on top of my commit (d37115f4b998b2780705f837137865d11a741f64) in this pull request:

diff --git a/Lib/test/test_unicodedata.py b/Lib/test/test_unicodedata.py
index 275a2ed747..757011a72c 100644
--- a/Lib/test/test_unicodedata.py
+++ b/Lib/test/test_unicodedata.py
@@ -322,7 +322,7 @@ class NormalizationTest(unittest.TestCase):
 
     def test_normalization(self):
         TESTDATAFILE = "NormalizationTest.txt"
-        TESTDATAURL = f"http://www.pythontest.net/unicode/{unicodedata.unidata_version}/{TESTDATAFILE}"
+        TESTDATAURL = f"http://www.pythontest.net/does_not_exist.txt"
 
         # Hit the exception early
         try:

The normalization test fails with: (I tested on Linux instead of Android this time - should be irrelevant)

$ ./python -m test -v test_unicodedata -m test_normalization -uall 
== CPython 3.10.0a0 (heads/[bpo-41391](https://bugs.python.org/issue41391)-dirty:d37115f4b9, Aug 4 2020, 22:29:10) [GCC 10.1.0]
== Linux-5.7.11-arch1-1-x86_64-with-glibc2.31 little-endian
== cwd: /home/yen/var/local/Computer/python/cpython/build/test_python_15442æ
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 0.42 Run tests sequentially
0:00:00 load avg: 0.42 [1/1] test_unicodedata
test_normalization (test.test_unicodedata.NormalizationTest) ...        fetching http://www.pythontest.net/does_not_exist.txt ...
FAIL

======================================================================
FAIL: test_normalization (test.test_unicodedata.NormalizationTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yen/var/local/Computer/python/cpython/Lib/test/test_unicodedata.py", line 330, in test_normalization
    testdata = open_urlresource(TESTDATAURL, encoding="utf-8",
urllib.error.HTTPError: HTTP Error 404: Not Found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/yen/var/local/Computer/python/cpython/Lib/test/test_unicodedata.py", line 336, in test_normalization
    self.fail(f"Could not retrieve {TESTDATAURL}")
AssertionError: Could not retrieve http://www.pythontest.net/does_not_exist.txt

----------------------------------------------------------------------

Ran 1 test in 0.524s

FAILED (failures=1)
test test_unicodedata failed
test_unicodedata failed

== Tests result: FAILURE ==

1 test failed:
    test_unicodedata

Total duration: 599 ms
Tests result: FAILURE

[yan12125]

Oops, I didn't notice there is already a simpler fix pushed after I created this pull request: #24650.