Skip to content

gh-95027: Ensure test runner uses utf-8:surrogateescape for communicating with subprocesses #96669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

gh-95027: Ensure test runner uses utf-8:surrogateescape for communicating with subprocesses #96669

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Lib/test/libregrtest/main.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -681,6 +681,10 @@ def main(self, tests=None, **kwargs):

self.fix_umask()

# We may have been launched with a certain IO encoding, but we do not
# want children to inherit it, so clear it out now.
os.unsetenv('PYTHONIOENCODING')

if self.ns.cleanup:
self.cleanup()
sys.exit(0)
Expand Down
4 changes: 3 additions & 1 deletion Lib/test/libregrtest/runtest_mp.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,9 @@ def run_test_in_subprocess(testname: str, ns: Namespace, tmp_dir: str, stdout_fh
'-m', 'test.regrtest',
'--worker-args', worker_args]


env = dict(os.environ)
env['PYTHONIOENCODING'] = 'utf-8:surrogateescape'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer to test the Python "default" encoding, and so fix code which decodes test stdout instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Python test suite worked well with any encoding for years without having to force an encoding, it was broken recently to support WASM.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then remove the encoding='utf-8' from the code that reads from the file.

But since this isn't testing, and is just IPC between test runners, I'd prefer to keep full fidelity in the data.

if tmp_dir is not None:
env['TMPDIR'] = tmp_dir
env['TEMP'] = tmp_dir
Expand Down Expand Up @@ -270,7 +272,7 @@ def _runtest(self, test_name: str) -> MultiprocessResult:
# gh-94026: Write stdout+stderr to a tempfile as workaround for
# non-blocking pipes on Emscripten with NodeJS.
with tempfile.TemporaryFile(
'w+', encoding=sys.stdout.encoding
'w+', encoding=sys.stdout.encoding, errors="surrogateescape",
) as stdout_fh:
# gh-93353: Check for leaked temporary files in the parent process,
# since the deletion of temporary files can happen late during
Expand Down
2 changes: 2 additions & 0 deletions Misc/NEWS.d/next/Tests/2022-09-07-23-44-06.gh-issue-95027.H5xs9o.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Ensures test runner uses UTF-8 and surrogateescape encoding when
communicating between multiple processes.