Additional warning text in pip output breaks Mu's processing of `pip list` during venv creation/verification

[aroberge]

mu.log
REMEMBER TO ATTACH YOUR LOG FILE

Version: 1.1.0b7
Locale: en_US
Platform: Windows 10 10.0.19043 AMD64

Traceback (most recent call last):
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\app.py", line 159, in run
    raise ex
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\app.py", line 146, in run
    venv.ensure_and_create(self.display_text)
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 621, in ensure_and_create
    self.recreate()
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 566, in recreate
    self.create()
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 773, in create
    self.register_baseline_packages()
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 892, in register_baseline_packages
    packages = list(self.pip.installed())
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 351, in installed
    name, version = line.split()[:2]
ValueError: not enough values to unpack (expected 2, got 1)

[ntoll]

Hmm... is this installed via pip or the Windows installer..?

[aroberge]

Windows installer.

[ntoll]

Hmm... was this a "fresh" install (i.e. you uninstalled the previous version before this latest one)..?

(Apologies for all the q's, but want to pin down precisely the circumstances so we can try to recreate - needless to say, we have tested the installer on Windows and "it worked for us". Trying to work out what might be different about your machine compared with, say, my Windows 10 machine.)

[ntoll]

(It goes without saying we really appreciate your patience and taking the time to report these details.)

[aroberge]

No problem about additional questions! I was just going to add that I still have the previous beta installed as I thought it might be the cause of the problem.

I'm about to start a 3 hour long meeting at work; after that, I'll remove the b6 version, try again, and will report.

Sorry to leave you waiting.

[ntoll]

No problem at all and best of luck with the mega-meeting (poor old you!).

[aroberge]

Not going well for me ...

I removed b6 and b7, and reinstalled. Another crash :(

REMEMBER TO ATTACH YOUR LOG FILE

Version: 1.1.0b7
Locale: en_US
Platform: Windows 10 10.0.19043 AMD64

Traceback (most recent call last):
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\app.py", line 159, in run
    raise ex
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\app.py", line 146, in run
    venv.ensure_and_create(self.display_text)
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 621, in ensure_and_create
    self.recreate()
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 553, in recreate
    _, user_packages = self.installed_packages()
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 928, in installed_packages
    for package, version in self.pip.installed():
  File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 351, in installed
    name, version = line.split()[:2]
ValueError: not enough values to unpack (expected 2, got 1)

mu.log

[aroberge]

The problem does not appear to be with Mu but rather with something on my computer that has changed :(

1. Removed b7 again using add/remove programs
2. Manually removed mu folder located at c:/users/andre/AppData/Local/python/mu and all its content
3. Attempted to reinstall b7 using msi: Crashed: identical traceback as previous two.
4. Removed b7 again using add/remove programs.
5. Attempted to reinstall b6 from previously saved msi: crashed !

I will try on a different computer.

mu.log

**REMEMBER TO ATTACH YOUR LOG FILE**

Version: 1.1.0.beta.6
Locale: en_US
Platform: Windows 10 10.0.19043 AMD64

Traceback (most recent call last):
File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\app.py", line 159, in run
raise ex
File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\app.py", line 146, in run
venv.ensure_and_create(self.display_text)
File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 621, in ensure_and_create
self.recreate()
File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 553, in recreate
_, user_packages = self.installed_packages()
File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 923, in installed_packages
for package, version in self.pip.installed():
File "C:\Users\andre\AppData\Local\Programs\Mu Editor\Python\lib\site-packages\mu\virtual_environment.py", line 351, in installed
name, version = line.split()[:2]
ValueError: not enough values to unpack (expected 2, got 1)

[aroberge]

I rebooted my computer, tried to reinstall ... crash.

I tried on my newer computer, and it worked. 💯

Both computers are Windows 10 Pro. The older one took an awful long time to install Mu - the antimalware program was using about 15% of the CPU.

As you do not have access to my computer, I tried the following experiment.

Just above the line where the error occurs, I wrote the following:

            with open("c:\\users\\andre\\mu_log.txt", "a") as f:
                f.write(line+"\n")

Here's the output just before the crash:

C:\Users\andre\AppData\Local\python\mu\mu_venv-38-20211222-130320\lib\site-packages\pip\_vendor\packaging\version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(

I figured that DeprecationWarnings were not expected as an output ... so, I changed this to the following:

        for line in iterlines:
            #
            # Some lines have a third location element
            #
            with open("c:\\users\\andre\\mu_log.txt", "a") as f:
                f.write(line+"\n")
            if "DeprecationWarning" in line:
                continue
            try:
                name, version = line.split()[:2]
                yield name, version
            except ValueError:
                pass

And it worked! Here's the content of mu_log.txt (after I erased the first version).

C:\Users\andre\AppData\Local\python\mu\mu_venv-38-20211222-130320\lib\site-packages\pip\_vendor\packaging\version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
Package           Version
----------------- -------
backcall          0.2.0
bitstring         3.1.9
cffi              1.15.0
click             8.0.3
colorama          0.4.4
cryptography      36.0.1
decorator         5.1.0
ecdsa             0.17.0
entrypoints       0.3
esptool           3.2
Flask             1.1.2
ipykernel         5.5.6
ipython           7.30.1
ipython-genutils  0.2.0
itsdangerous      2.0.1
jedi              0.18.1
Jinja2            3.0.3
jupyter-client    7.1.0
jupyter-core      4.9.1
MarkupSafe        2.0.1
matplotlib-inline 0.1.3
nest-asyncio      1.5.4
numpy             1.21.5
parso             0.8.3
pgzero            1.2.1
pickleshare       0.7.5
pip               21.3.1
prompt-toolkit    3.0.24
pycparser         2.21
pygame            2.1.0
Pygments          2.10.0
pyserial          3.5
python-dateutil   2.8.2
pywin32           303
pyzmq             22.3.0
reedsolo          1.5.4
setuptools        60.0.3
six               1.16.0
tornado           6.1
traitlets         5.1.1
wcwidth           0.2.5
Werkzeug          2.0.2
wheel             0.36.2
C:\Users\andre\AppData\Local\python\mu\mu_venv-38-20211222-140132\lib\site-packages\pip\_vendor\packaging\version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
Package           Version
----------------- -------
backcall          0.2.0
bitstring         3.1.9
cffi              1.15.0
click             8.0.3
colorama          0.4.4
cryptography      36.0.1
decorator         5.1.0
ecdsa             0.17.0
entrypoints       0.3
esptool           3.2
Flask             1.1.2
ipykernel         5.5.6
ipython           7.30.1
ipython-genutils  0.2.0
itsdangerous      2.0.1
jedi              0.18.1
Jinja2            3.0.3
jupyter-client    7.1.0
jupyter-core      4.9.1
MarkupSafe        2.0.1
matplotlib-inline 0.1.3
nest-asyncio      1.5.4
numpy             1.21.5
parso             0.8.3
pgzero            1.2.1
pickleshare       0.7.5
pip               21.3.1
prompt-toolkit    3.0.24
pycparser         2.21
pygame            2.1.0
Pygments          2.10.0
pyserial          3.5
python-dateutil   2.8.2
pywin32           303
pyzmq             22.3.0
reedsolo          1.5.4
setuptools        60.0.3
six               1.16.0
tornado           6.1
traitlets         5.1.1
wcwidth           0.2.5
Werkzeug          2.0.2
wheel             0.36.2
C:\Users\andre\AppData\Local\python\mu\mu_venv-38-20211222-140329\lib\site-packages\pip\_vendor\packaging\version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
Package           Version
----------------- -------
backcall          0.2.0
bitstring         3.1.9
cffi              1.15.0
click             8.0.3
colorama          0.4.4
cryptography      36.0.1
decorator         5.1.0
ecdsa             0.17.0
entrypoints       0.3
esptool           3.2
Flask             1.1.2
ipykernel         5.5.6
ipython           7.30.1
ipython-genutils  0.2.0
itsdangerous      2.0.1
jedi              0.18.1
Jinja2            3.0.3
jupyter-client    7.1.0
jupyter-core      4.9.1
MarkupSafe        2.0.1
matplotlib-inline 0.1.3
nest-asyncio      1.5.4
numpy             1.21.5
parso             0.8.3
pgzero            1.2.1
pickleshare       0.7.5
pip               21.3.1
prompt-toolkit    3.0.24
pycparser         2.21
pygame            2.1.0
Pygments          2.10.0
pyserial          3.5
python-dateutil   2.8.2
pywin32           303
pyzmq             22.3.0
reedsolo          1.5.4
setuptools        60.0.3
six               1.16.0
tornado           6.1
traitlets         5.1.1
wcwidth           0.2.5
Werkzeug          2.0.2
wheel             0.36.2

I am a bit surprised to see the output in triplicate.

Deleting mu_log.txt and restarting Mu does not generate any content in mu_log.txt.

Let me know if there's any information I might be able to get that could be useful.

[aroberge]

I forgot. Just as the last install succeeded, the splash screen seemed to hang for a long time and I noticed what I thought was a weird message about quarantine ... I had enough time to casually take a screen capture, in case it could be of some use:

[carlosperate]

Thanks for the report and detailed analysis!

I think the problem is that the output of pip list has more content than expected (all those warning messages on top) and it tries to process the wrong line as a package line.

Basically this bit of code throws away the first two lines from pip list and then splits the following lines (where we have things like backcall 0.2.0) to separate package name from version:

mu/mu/virtual_environment.py

Lines 333 to 352 in 5e2253e

	lines = self.list().splitlines()
	iterlines = iter(lines)
	#
	# The first two lines are headers
	#
	try:
	next(iterlines)
	next(iterlines)
	#
	# cf https://lgtm.com/rules/11000086/
	#
	except StopIteration:
	raise VirtualEnvironmentError("Unable to parse installed packages")
	for line in iterlines:
	#
	# Some lines have a third location element
	#
	name, version = line.split()[:2]
	yield name, version

This works fine for a normal pip list output:

Package           Version
----------------- -------
backcall          0.2.0
...

But the log captured this as the output of pip list:

C:\Users\andre\AppData\Local\python\mu\mu_venv-38-20211222-093341\lib\site-packages\pip\_vendor\packaging\version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
C:\Users\andre\AppData\Local\python\mu\mu_venv-38-20211222-093341\lib\site-packages\pip\_vendor\packaging\version.py:111: DeprecationWarning: Creating a LegacyVersion has been deprecated and will be removed in the next major release
  warnings.warn(
Package           Version
----------------- -------
backcall          0.2.0

I think rather than next(iterlines) twice, we should iter checking each line until we find the Package... and -------... lines.