Implementation of the PEP 538: coerce C locale to C.utf-8

[JanNiklasHasse]

BPO	28180
Nosy	@malemburg, @warsaw, @ronaldoussoren, @ncoghlan, @vstinner, @ned-deily, @mcepl, @ezio-melotti, @bitdancer, @methane, @4kir4, @xdegaye, @yan12125, @Vgr255
PRs	bpo-28180: Implementation for PEP 538 #659 bpo-28180: assume UTF-8 for Mac OS X PEP 538 tests #2130 bpo-28180: Fix test_capi.test_forced_io_encoding() #2155 bpo-28180: Standard stream & FS encoding differ on Mac OS X #2208 bpo-28180: Fix the implementation of PEP 538 on Android #4334
Dependencies	bpo-30565: PEP 538: silence locale coercion and compatibility warnings by default? bpo-30635: Leak in test_c_locale_coercion bpo-30647: CODESET error on AMD64 FreeBSD 10.x Shared 3.x caused by the PEP 538
Files	fedora-cpython-force-c-utf-8.diff: Downstream patch currently proposed for Fedora 26 fedora-cpython-PYTHONALLOWCLOCALE.diff: Draft Fedora 26 patch as at 2016-12-18 pep538_coerce_legacy_c_locale.diff: Initial patch for PEP 538 (targeting 3.7) pep538_coerce_legacy_c_locale_v2.diff: Add test cases for handling of unknown locales pep538-check-click.sh: Utility script to check click's behaviour in a PEP 538 patched CPython pep538_coerce_legacy_c_locale_v3.diff: Refactor PEP 538 test cases to cover no locale setting, C locale, POSIX locale and unknown locale android_setlocale.patch pep538_coerce_legacy_c_locale.patch: Ufinished attempt to port this patch to Python 3.4

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2018-03-29.01:23:38.218>
created_at = <Date 2016-09-16.11:17:02.590>
labels = ['type-bug', '3.7', 'expert-unicode']
title = 'Implementation of the PEP 538: coerce C locale to C.utf-8'
updated_at = <Date 2020-03-22.16:25:00.648>
user = 'https://bugs.python.org/JanNiklasHasse'

bugs.python.org fields:

activity = <Date 2020-03-22.16:25:00.648>
actor = 'mcepl'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2018-03-29.01:23:38.218>
closer = 'ncoghlan'
components = ['Unicode']
creation = <Date 2016-09-16.11:17:02.590>
creator = 'Jan Niklas Hasse'
dependencies = ['30565', '30635', '30647']
files = ['45907', '45951', '46059', '46121', '46190', '46205', '46329', '48991']
hgrepos = []
issue_num = 28180
keywords = ['patch']
message_count = 89.0
messages = ['276693', '276694', '276707', '276709', '276722', '276729', '277273', '277274', '282964', '282965', '282970', '282971', '282972', '282977', '282978', '282984', '283244', '283408', '283409', '283469', '283471', '283482', '283495', '283515', '283543', '283732', '284150', '284170', '284176', '284537', '284605', '284620', '284621', '284631', '284641', '284647', '284697', '284716', '284718', '284719', '284720', '284722', '284725', '284729', '284736', '284742', '284747', '284764', '284782', '284794', '284795', '284799', '284882', '284884', '284886', '284887', '284900', '284908', '284943', '284952', '285735', '286001', '289002', '289534', '295121', '295683', '295688', '295698', '295710', '295713', '295722', '295871', '295872', '295875', '295885', '295913', '295914', '296064', '296075', '296077', '305850', '306108', '314627', '364740', '364760', '364767', '364770', '364804', '364810']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'ronaldoussoren', 'ncoghlan', 'vstinner', 'ned.deily', 'mcepl', 'ezio.melotti', 'r.david.murray', 'methane', 'akira', 'Sworddragon', 'xdegaye', 'yan12125', 'abarry', 'Jan Niklas Hasse']
pr_nums = ['659', '2130', '2155', '2208', '4334']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28180'
versions = ['Python 3.7']

[JanNiklasHasse]

Working with Docker I often end up with an environment where the locale isn't correctly set. In these cases it would be great if sys.getfilesystemencoding() could default to 'utf-8' instead of 'ascii', as it's the encoding of the future and ascii is a subset of it anyway.

Related: http://bugs.python.org/issue19846

[Vgr255]

This is a duplicate of bpo-27781.

[vstinner]

This is a duplicate of bpo-27781.

bpo-27781 is specific to Windows. I'm not sure that it's the base in this issue. So I reopen the issue.

@jan Niklas Hasse: What is your OS?

I proposed to add "-X utf8" command line option for UNIX to force utf8 encoding. Would it work for you?

[JanNiklasHasse]

Unfortunately no, as this would mean I'll have to change all my python invocations in my scripts and it wouldn't work for executable files with

#!/usr/bin/env python3

would it?

[bitdancer]

I thought we "fixed" this by using surrogate escape when the locale was ASCII? We certainly have discussed changing the default and posix and so far have decided not to (someday that will change...is this someday already?)

[vstinner]

is this someday already?)

Not yet :-)

[JanNiklasHasse]

Why not?

[methane]

I want locale free Python which behaves like on C.UTF-8 locale.
(stdio encoding, preferred encoding, weekday in _strptime._strptime,
and more maybe)

But Python 3.6 is feature freeze already >_<;;

[ncoghlan]

I think we're genuinely getting to the point now where the majority of "LANG=C" cases are misconfigurations rather than intended behaviour. We're also to the point where:

• on Mac OS X, binary system interfaces have been handled as UTF-8 by default since 3.0
• on Windows, as of 3.6, the OS native binary system interfaces are now bypassed entirely in favour of transcoding from UTF-8 to UTF-16-LE

So I think for Python 3.7 it makes sense to do the following on other *nix systems:

• very early in CPython startup (even before argument processing), if the detected locale is "C", force it to "C.UTF-8" if possible, and print a warning either way
• add a PYTHONKEEPASCIILOCALE environment variable to turn that behaviour off

I do think we actually want to *change* the C level locale in the process though, as otherwise we can expect to see weird interactions where CPython and extension modules disagree about the default text encoding.

[ncoghlan]

Note also that if we say we're going to do this for 3.7, *and* go ahead and implement it, then distros may be more inclined to incorporate the same behavioural changes into distro-provided releases of 3.6, providing real world testing of the concept before we make it the default behaviour.