msgfmt cannot cope with BOM - improve error message

[Cito]

BPO	1697943
Nosy	@loewis, @rhettinger, @Cito, @vstinner, @merwok, @serhiy-storchaka
Files	msgfmt.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/loewis'
closed_at = None
created_at = <Date 2007-04-10.20:58:04.000>
labels = ['type-bug', 'expert-unicode', '3.11']
title = 'msgfmt cannot cope with BOM - improve error message'
updated_at = <Date 2021-04-22.15:26:29.553>
user = 'https://github.com/Cito'

bugs.python.org fields:

activity = <Date 2021-04-22.15:26:29.553>
actor = 'iritkatriel'
assignee = 'loewis'
closed = False
closed_date = None
closer = None
components = ['Demos and Tools', 'Unicode']
creation = <Date 2007-04-10.20:58:04.000>
creator = 'cito'
dependencies = []
files = ['2348']
hgrepos = []
issue_num = 1697943
keywords = ['patch']
message_count = 9.0
messages = ['31755', '31756', '31757', '31758', '70042', '125940', '125941', '290519', '290524']
nosy_count = 6.0
nosy_names = ['loewis', 'rhettinger', 'cito', 'vstinner', 'eric.araujo', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue1697943'
versions = ['Python 3.11']

Linked PRs

• gh-44827: Improve error if BOM on first line of .po file #130187

[Cito]

If a .po file has a BOM (byte order mark) at the beginning, as is often the case for utf-8 files created on Windows, msgfmt.py complines about a syntax error.

The attached patch fixes this problem.

[rhettinger]

Martin, is this your code?

[loewis]

It's my code, but I will need to establish first whether it's a bug. That depends on what the PO specification says, and, if is it silent on the matter, what GNU gettext does.

[Cito]

It may well be that GNU gettext also chokes on a BOM, because they aren't used under Linux. But I think as a Python tool it should be more Windows-tolerant. The annoying thing is that you get a syntax error, but everything looks right because the BOM is usually invisible. Such error messages are really frustrating. Either the BOM should be silently ignored (as in the patch) or the users should get a friendly error message asking them to save the file without BOM. If GNU behaves badly to Windows users, that's not an excuse to do the same. They are already suffering enough because of their (or their bosses') bad choice of OS ;-)

[Cito]

Small improvement of the patch: Instead of hardcoding the BOM as
'\xef\xbb\xbf', we should use codecs.BOM_UTF8.

[vstinner]

Extract of the Unicode standard: "Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature".

See also the following section explaing issues with UTF-8 BOM:
http://en.wikipedia.org/wiki/Byte_order_mark#UTF-8

I agree that Python should handle (UTF-8) BOM to read a CSV file (bpo-7185), because the file format is common on Windows.

But msgfmt is an UNIX tool: I would expect that Python behaves like the original msgfmt tool, fail with a fatal error on the BOM "invisible character". How do you explain to a user msgfmt fails but not msgfmt.py?

About the patch: *ignore* the BOM is not a good idea. The BOM announces the encoding (eg. UTF-8): if a Content-Type header announces another encoding, you should raise an error.

[vstinner]

See also issue bpo-7651: "Python3: guess text file charset using the BOM".

[serhiy-storchaka]

Corresponding GNU gettext issue [1] was closed as "Not a Bug".

[1] https://savannah.gnu.org/bugs/?18345