Source files are always read as UTF-8, so valid Python that CPython runs fine is reported as unparseable.
Two cases:
PEP 263 coding cookie — a file declaring another codec cannot be read at all, and surfaces as SIG902 with the critical exit status:
# -*- coding: latin-1 -*-
# café
def function(param: int) -> None:
"""Summary.
:param param: A description.
"""UTF-8 BOM — the byte-order mark some editors prepend leaks into the source and surfaces as SIG901 invalid-syntax.
Expected: no error in either case. docsig should follow the Python tokenizer's own rules (tokenize.detect_encoding, which handles both the coding cookie and the BOM) when deciding how to decode a file, so anything CPython can run can be read. Files that genuinely cannot be decoded should still surface as a unicode decode error.
Source files are always read as UTF-8, so valid Python that CPython runs fine is reported as unparseable.
Two cases:
PEP 263 coding cookie — a file declaring another codec cannot be read at all, and surfaces as SIG902 with the critical exit status:
UTF-8 BOM — the byte-order mark some editors prepend leaks into the source and surfaces as SIG901 invalid-syntax.
Expected: no error in either case. docsig should follow the Python tokenizer's own rules (
tokenize.detect_encoding, which handles both the coding cookie and the BOM) when deciding how to decode a file, so anything CPython can run can be read. Files that genuinely cannot be decoded should still surface as a unicode decode error.