Dependency errors when using python 3.9 and 3.8

[mjt91]

Issue

When using python version 3.9: For me, all nox sessions where install_with_constrains is used with python version 3.9 fail.

For instance running the tests session works fine, till chapter 4. After adding more nox sessions and installing more dependencies I started to see multiple dependencies for different python versions (I am following the tutorial using python 3.9 and 3.8 respectively).

After some research, it seems this problem is related to Duplicate dependency export #1970

Console Output

CLICK ME

nox > Running session tests-3.9
nox > Re-using existing virtual environment at .nox/tests-3-9.
nox > poetry install --no-dev
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 11 removals

  • Removing coverage (5.3)
  • Removing iniconfig (1.1.1)
  • Removing packaging (20.4)
  • Removing pluggy (0.13.1)
  • Removing py (1.9.0)
  • Removing pyparsing (2.4.7)
  • Removing pytest (6.1.1)
  • Removing pytest-cov (2.10.1)
  • Removing pytest-mock (3.3.1)
  • Removing six (1.15.0)
  • Removing toml (0.10.1)
  • Installing mypy-extensions (0.4.3)
  • Installing typing-extensions (3.7.4.3)

Installing the current project: hypermodern-bersten (0.1.0)
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmppd82bei6
nox > pip install --constraint=/tmp/tmppd82bei6 coverage[toml] pytest pytest-cov pytest-mock
nox > Command pip install --constraint=/tmp/tmppd82bei6 coverage[toml] pytest pytest-cov pytest-mock failed with exit code 1:
Ignoring atomicwrites: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.4.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring attrs: markers 'python_version == "3.8" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version == "3.8"' don't match your environment
Ignoring colorama: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") and platform_system == "Windows" or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.5.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") and platform_system == "Windows"' don't match your environment
Ignoring decorator: markers 'python_version == "3.8" and python_full_version < "3.0.0" and python_full_version >= "2.7.0" or python_version == "3.8" and python_full_version >= "3.2.0"' don't match your environment
Ignoring importlab: markers 'python_version == "3.8" and python_full_version >= "2.7.0"' don't match your environment
Ignoring networkx: markers 'python_version == "3.8" and python_full_version >= "2.7.0"' don't match your environment
Ignoring ninja: markers 'python_version == "3.8"' don't match your environment
Ignoring pytype: markers 'python_version == "3.8"' don't match your environment
Ignoring pyyaml: markers 'python_version == "3.8" and python_full_version < "3.0.0" or python_version == "3.8" and python_full_version >= "3.5.0"' don't match your environment
Ignoring six: markers 'python_version == "3.8" and python_full_version < "3.0.0" and python_full_version >= "2.7.0" or python_version == "3.8" and python_full_version >= "3.4.0"' don't match your environment
Ignoring typed-ast: markers 'python_version == "3.8"' don't match your environment
Collecting coverage==5.3
  Using cached coverage-5.3-cp39-cp39-manylinux1_x86_64.whl (228 kB)
Collecting pytest-cov==2.10.1
  Using cached pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)
Collecting pytest-mock==3.3.1
  Using cached pytest_mock-3.3.1-py3-none-any.whl (11 kB)
Collecting pytest==6.1.1
  Using cached pytest-6.1.1-py3-none-any.whl (272 kB)
Collecting toml==0.10.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Requirement already satisfied: attrs>=17.4.0 in ./.nox/tests-3-9/lib/python3.9/site-packages (from pytest==6.1.1->-c /tmp/tmppd82bei6 (line 176)) (20.2.0)
Collecting pluggy==0.13.1
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting iniconfig==1.1.1
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging==20.4
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting py==1.9.0
  Using cached py-1.9.0-py2.py3-none-any.whl (99 kB)
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting six
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    six from https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl#sha256=8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced (from packaging==20.4->-c /tmp/tmppd82bei6 (line 146))
nox > Session tests-3.9 failed.

[mjt91]

hypermodern-python/noxfile.py

Lines 30 to 39 in 400386b

	with tempfile.NamedTemporaryFile() as requirements:
	session.run(
	"poetry",
	"export",
	"--dev",
	"--format=requirements.txt",
	f"--output={requirements.name}",
	external=True,
	)
	session.install(f"--constraint={requirements.name}", *args, **kwargs)

I fixed this problem by adding --without-hashes parameter to the install_with_constraints function in noxfile.

[ToddG]

I am running into this issue with python3.8 (though I do not run into this with python3.7 and have not yet tried python3.9). I submitted a bug to pytype suggesting they add explicit version pinning to their requirements.txt file:

google/pytype#731

If you add the --without-hashes then doesn't that remove the guarantee that the install_with_constraints function is providing?

Also, I created my own seed repo as I did not see that @cjolowicz had created a seed repo of his own. I'll look at that and then potentially deprecate mine.

[allesmi]

I have the same problem with python 3.8 and 3.9. Only nox sessions with python 3.7 do not fail with variations of the above console output. Other packages that are missing hashes in the output of poetry export are attrs, pyYAML, and typed-ast, and probably others.

Is this behavior caused by poetry or pip or pyenv or python?

While adding the argument --without-hashes to the export commands helps to avoid the error this workaround of ignoring hashes is not the right way to go. Is the only permanent solution to nudge package maintainers to pin versions explicitly like @ToddG did for pytype? Both approaches do not appear very hypermodern to me.

[cjolowicz]

Could you try downgrading Poetry to 1.1.2, or if that doesn't help, 1.0.10?

[cjolowicz]

In the meantime, please refrain from asking maintainers of other projects to pin their requirements. This is based on a misunderstanding: Even if pytype pinned their requirements.txt, this would not affect you, because those pins would not end up in the package metadata. The pytype dependencies you see are what pytype declares in their setup.cfg. And you would not want those to be pinned, otherwise chances would be very slim that you could add several development dependencies alongside each other.

[allesmi]

After downgrading poetry to 1.1.2 the nox sessions for lint, mypy, and tests under python 3.8 run successfully.

Edit: The above issue also occurs for poetry 1.1.3

[wanderrful]

After downgrading poetry to 1.1.2 the nox sessions for lint, mypy, and tests under python 3.8 run successfully.

Edit: The above issue also occurs for poetry 1.1.3

Sadly, this did not work for me. Error message is effectively the same, though for all Nox sessions. Tried using Poetry 1.0.10, 1.1.2, and 1.1.4 (latest at time of writing). Also tried using different combinations of recent Pip versions as well as the most recent one.

[ToddG]

@wanderrful

Here is what I am currently using:

def install_with_constraints(session: Session, *args: str, **kwargs: Any) -> None:
    """Install packages constrained by Poetry's lock file.

    This function is a wrapper for nox.sessions.Session.install. It
    invokes pip to install packages inside of the session's virtualenv.
    Additionally, pip is passed a constraints file generated from
    Poetry's lock file, to ensure that the packages are pinned to the
    versions specified in poetry.lock. This allows you to manage the
    packages as Poetry development dependencies.

    Arguments:
        session: The Session object.
        args: Command-line arguments for pip.
        kwargs: Additional keyword arguments for Session.install.
    """
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            "--without-hashes",
            f"--output={requirements.name}",
            external=True,
        )
        session.install(f"--constraint={requirements.name}", *args, **kwargs)

Then using it within nox sessions as:

@nox.session(python=_versions)
def lint(session: Session) -> None:
    """Run the code linters."""
    args = session.posargs or locations
    install_with_constraints(
        session,
        "darglint",
        "flake8",
        "flake8-annotations",
        "flake8-black",
        "flake8-docstrings",
        "flake8-isort",
        "flake8-rst-docstrings",
        "flake8_sphinx_links",
    )
    session.run("flake8", *args)

I hope that helps.

[wanderrful]

Yeah I have the same setup pretty much. I've checked the exported requirements.txt files for the dependencies it complains about and it looks fine to me. So, it makes me think that PIP itself is somehow wrongly parsing the exported requirements.txt file, which would be Poetry's fault... but if it's Poetry's fault then you'd think that using an earlier version in CICD builds would fix the problem. However, again, Poetry's different versions don't fix the issue.

It's pretty strange and makes me wonder if the DockerHub python slim images I'm using are themselves somehow the problem... but they haven't been updated for 7 months, so maybe it's not at a lower layer?

[wanderrful]

Update- I got it!

I was on the right track, with my previous comment above. However, instead of looking down to the Docker image layer, I had forgotten that there is one last layer before we get to Docker, which is virtualenv!

Manually pinning virtualenv to version 20.0.26 (current version is 20.2.2) fixes the issue. Not sure if it's just this specific current version that introduces the issue, though.

[cjolowicz]

Glad you found a workaround. I would suspect that this is triggered by changes to constraints file handling in pip's new resolver. Pip is bundled with virtualenv. You could then also "solve" this by pinning pip to a version before the new resolver became the default, setting the environment variable VIRTUALENV_PIP to 20.2.4

[shadycuz]

We also were having this issue ^.

Should an issue be raised with pip @cjolowicz or is this not a bug, just something we are doing wrong?

It seems like it's a bug if the error is the hashes are not present, but they are.

Edit: based on the discussion pypa/pip#8792 its a design decision and not a bug.