Decoding error on license file on macOS

[johnthagen]

Same issue as #35 and #37 but on macOS with a standard Python 3 install, not in a docker container.

Tested on 1.16.1.

$ pip-licenses --from=mixed --with-system --format=rst --with-license-file
Traceback (most recent call last):
  File "/Users/user/proj/.tox/licenses/bin/pip-licenses", line 8, in <module>
    sys.exit(main())
  File "/Users/user/proj/.tox/licenses/site-packages/piplicenses.py", line 638, in main
    print(output_string)
  File "/Users/user/proj/.tox/licenses/lib-python/3/encodings/ascii.py", line 22, in encode
    return codecs.ascii_encode(input, self.errors)[0]
UnicodeEncodeError: 'ascii' codec can't encode character '\xa9' in position 167881: ordinal not in range(128)

Python prompt for reference:

$ python
Python 3.6.9 (5da45ced70e5, Oct 09 2019, 19:13:07)
[PyPy 7.2.0 with GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

[johnthagen]

The package whose license file is failing is Pillow == 6.2.1

[raimon49]

Thanks for the report. I also try it on my macOS.

[raimon49]

@jameslamb It didn't reproduce on my macOS.

However, it was reproduced when executed as follows.

# raise Same error you reported 'UnicodeEncodeError'
$ env LANG=US-ASCII LC_CTYPE=US-ASCII pip-licenses --from=mixed --with-system --format=rst --with-license-file

From this fact, I am suspicious of LANG and LC_CTYPE of tox environment.

Write passenv in your tox.ini file and run it again.

[testenv]
passenv= LANG LC_CTYPE

[johnthagen]

[testenv]
passenv= LC_CTYPE

fixed it, thanks!

Still strange though. I suppose that it's because tox doesn't bring in any ENV variables by default. But, then again lots of Python programs can read Unicode files from within tox, so I wonder if there is a way to make pip-licenses work out of the box? It seems like if it's reading arbitrary license files from PyPI it shouldn't really need to assume the user has a certain locale?

Hmm, not sure, but really appreciate the fix!

[raimon49]

I add to the documentation as a frequently asked question about UnicodeEncodeError.

Your report was very helpful. Thanks a lot!

[reactive-firewall]

Important

Starting with PEP 263 in 2001, UTF-8 replaces
ascii as the default, providing strong historical support for expecting UTF-8.
Further, after this issue was resolved, PEP-686
has standardized this in relation to the open function's default, making the fix here
no longer even necessary.
TL;DR My rational for commenting on a closed issue is: this issue is referenced in the
pip-licenses documentation, and the general confusion of the discussion here about
defaulting to UTF-8 as an encoding. I hope this helps someone in the future.

Note

This variation of #35 is related to PEP 597