ElementTree does not handle UTF-8 encoding

[ironpythonbot]

import tempfile
>>> from xml.etree.ElementTree import ElementTree
>>> xml = '\n\n'
>>> with tempfile.TemporaryFile() as f:
... f.write(bytes(xml, 'utf-8')) # use xml.encode('utf-8') in CPython 2.7
... f.flush()
... f.seek(0)
... tree = ElementTree(file=f)
... name = next(tree.iter()).get('name')
... print(repr(name))
... assert name == unichr(169)
...
u'\xc2\xa9'
Traceback (most recent call last):
File "", line 8, in
AssertionError

unichr(169) is the copyright sign "©" and is encoded in UTF-8 as b'\xc2\xa9' . The two-byte encoding is ignored by ElementTree and gets
interpreted as two separate characters.

Work Item Details

Original CodePlex Issue: Issue 35635
Status: Proposed
Reason Closed: Unassigned
Assigned to: Unassigned
Reported on: Oct 21 at 4:23 AM
Reported by: ysitu
Updated on: Nov 7 at 2:43 PM
Updated by: tcalmant

[slozier]

Repro code:

import tempfile
from xml.etree.ElementTree import ElementTree
xml = '<?xml version="1.0" encoding="UTF-8"?>\n<test name="' + unichr(169) + '"/>\n'
with tempfile.TemporaryFile() as f:
    f.write(xml.encode('utf-8'))
    f.flush()
    f.seek(0)
    tree = ElementTree(file=f)
    name = next(tree.iter()).get('name')
    print(repr(name))
    assert name == unichr(169)