importing pkg_resources fails

[lithp]

I believe that pkg_resouces is simply incompatible with pyoxidizer, but I've only been playing with pyoxidizer for a few hours so maybe someone here knows better. Is there a way to get pkg_resources to work? It's used in quite a few places by the tool I'm trying to package up!

If I init a new project and then make two changes:

[[packaging_rule]]
type = "pip-install-simple"
package = "setuptools>=36.2.0"

[[embedded_python_run]]
mode = "eval"
code = "import pkg_resources"

Then pyoxidizer run gives me a fun error:

Traceback (most recent call last):
  File "linecache", line 95, in updatecache
FileNotFoundError: [Errno 2] No such file or directory: 'pkg_resources._vendor.pyparsing'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "pkg_resources", line 84, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "pkg_resources.extern", line 43, in load_module
  File "pkg_resources._vendor.packaging.requirements", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "pkg_resources.extern", line 43, in load_module
  File "pkg_resources._vendor.pyparsing", line 4756, in <module>
  File "pkg_resources._vendor.pyparsing", line 1284, in setParseAction
  File "pkg_resources._vendor.pyparsing", line 1066, in _trim_arity
  File "pkg_resources._vendor.pyparsing", line 1050, in extract_stack
  File "traceback", line 211, in extract_stack
  File "traceback", line 363, in extract
  File "traceback", line 285, in line
  File "linecache", line 16, in getline
  File "linecache", line 47, in getlines
  File "linecache", line 103, in updatecache
  File "<frozen importlib._bootstrap_external>", line 566, in decode_source
AttributeError: 'memoryview' object has no attribute 'decode'
error: cargo run failed

Here's the relevant code:

https://github.com/python/cpython/blob/5e1400a6bcbb3350a6665176980a2b8343075c63/Lib/linecache.py#L94-L105

It looks like python is trying to stat the file. The stat fails because pyoxidizer is like that. Then, something deep inside importlib is attempting to use the memoryview which pyoxidizer passes in and failing, probably because it expects straight bytes?

[GonzRon]

I hit the following today:

Traceback (most recent call last):
  File "linecache", line 95, in updatecache
FileNotFoundError: [Errno 2] No such file or directory: 'pkg_resources._vendor.pyparsing'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "importlib.util", line 94, in find_spec
  File "rdt", line 23, in <module>
  File "Common.ET.envTracker", line 4, in <module>
  File "slumber", line 1, in <module>
  File "requests", line 112, in <module>
  File "requests.utils", line 26, in <module>
  File "requests._internal_utils", line 11, in <module>
  File "requests.compat", line 29, in <module>
  File "simplejson", line 114, in <module>
  File "simplejson.decoder", line 8, in <module>
  File "simplejson.scanner", line 11, in <module>
  File "simplejson.scanner", line 7, in _import_c_make_scanner
  File "simplejson._speedups", line 7, in <module>
  File "simplejson._speedups", line 3, in __bootstrap__
  File "pkg_resources", line 84, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "pkg_resources.extern", line 43, in load_module
  File "pkg_resources._vendor.packaging.requirements", line 9, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "pkg_resources.extern", line 43, in load_module
  File "pkg_resources._vendor.pyparsing", line 4756, in <module>
  File "pkg_resources._vendor.pyparsing", line 1284, in setParseAction
  File "pkg_resources._vendor.pyparsing", line 1066, in _trim_arity
  File "pkg_resources._vendor.pyparsing", line 1050, in extract_stack
  File "traceback", line 211, in extract_stack
  File "traceback", line 363, in extract
  File "traceback", line 285, in line
  File "linecache", line 16, in getline
  File "linecache", line 47, in getlines
  File "linecache", line 103, in updatecache
  File "<frozen importlib._bootstrap_external>", line 566, in decode_source
AttributeError: 'memoryview' object has no attribute 'decode'

Any suggestions or workarounds?

[lithp]

The best workaround right now is to not use pkg_resources @GonzRon

[anthrotype]

not using pkg_resources is not an option, if my app requires fs (pyfilesystem2) which in turn requires pkg_resources :(

[indygreg]

AttributeError: 'memoryview' object has no attribute 'decode' is an interesting error. What's likely happening here is the code path to get the Python module source doesn't like the fact that we are using a memoryview to represent source instead of bytes. This is a side-effect of PyOxidizer's in-memory module importing.

The code that is failing in Python's standard library is:

    import tokenize  # To avoid bootstrap issues.
    source_bytes_readline = _io.BytesIO(source_bytes).readline
    encoding = tokenize.detect_encoding(source_bytes_readline)
    newline_decoder = _io.IncrementalNewlineDecoder(None, True)
    return newline_decoder.decode(source_bytes.decode(encoding[0]))

and source_bytes here is a memoryview (as opposed to bytes).

We should be able to work around this by having our get_source() call into decode_source() with a bytes or bytearray instead of memoryview.

In other news, it is unfortunate that Python doesn't have a standard library type that supports 0-copy and exposes .decode(). We have bytearray, which is close. But the constructor for bytearray copies data into a new buffer. This is really unfortunate.

[indygreg]

I was able to import pkg_resources using the commit I just pushed.