Dependency errors when using python 3.9 and 3.8

[mjt91]

Issue

When using python version 3.9: For me, all nox sessions where install_with_constrains is used with python version 3.9 fail.

For instance running the tests session works fine, till chapter 4. After adding more nox sessions and installing more dependencies I started to see multiple dependencies for different python versions (I am following the tutorial using python 3.9 and 3.8 respectively).

After some research, it seems this problem is related to Duplicate dependency export #1970

Console Output

CLICK ME

nox > Running session tests-3.9
nox > Re-using existing virtual environment at .nox/tests-3-9.
nox > poetry install --no-dev
Installing dependencies from lock file

Package operations: 2 installs, 0 updates, 11 removals

  • Removing coverage (5.3)
  • Removing iniconfig (1.1.1)
  • Removing packaging (20.4)
  • Removing pluggy (0.13.1)
  • Removing py (1.9.0)
  • Removing pyparsing (2.4.7)
  • Removing pytest (6.1.1)
  • Removing pytest-cov (2.10.1)
  • Removing pytest-mock (3.3.1)
  • Removing six (1.15.0)
  • Removing toml (0.10.1)
  • Installing mypy-extensions (0.4.3)
  • Installing typing-extensions (3.7.4.3)

Installing the current project: hypermodern-bersten (0.1.0)
nox > poetry export --dev --format=requirements.txt --output=/tmp/tmppd82bei6
nox > pip install --constraint=/tmp/tmppd82bei6 coverage[toml] pytest pytest-cov pytest-mock
nox > Command pip install --constraint=/tmp/tmppd82bei6 coverage[toml] pytest pytest-cov pytest-mock failed with exit code 1:
Ignoring atomicwrites: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.4.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5")' don't match your environment
Ignoring attrs: markers 'python_version == "3.8" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version == "3.8"' don't match your environment
Ignoring colorama: markers 'python_version >= "3.5" and python_full_version < "3.0.0" and sys_platform == "win32" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") and platform_system == "Windows" or sys_platform == "win32" and python_version >= "3.5" and python_full_version >= "3.5.0" and (python_version >= "3.5" and python_full_version < "3.0.0" or python_full_version >= "3.5.0" and python_version >= "3.5") and platform_system == "Windows"' don't match your environment
Ignoring decorator: markers 'python_version == "3.8" and python_full_version < "3.0.0" and python_full_version >= "2.7.0" or python_version == "3.8" and python_full_version >= "3.2.0"' don't match your environment
Ignoring importlab: markers 'python_version == "3.8" and python_full_version >= "2.7.0"' don't match your environment
Ignoring networkx: markers 'python_version == "3.8" and python_full_version >= "2.7.0"' don't match your environment
Ignoring ninja: markers 'python_version == "3.8"' don't match your environment
Ignoring pytype: markers 'python_version == "3.8"' don't match your environment
Ignoring pyyaml: markers 'python_version == "3.8" and python_full_version < "3.0.0" or python_version == "3.8" and python_full_version >= "3.5.0"' don't match your environment
Ignoring six: markers 'python_version == "3.8" and python_full_version < "3.0.0" and python_full_version >= "2.7.0" or python_version == "3.8" and python_full_version >= "3.4.0"' don't match your environment
Ignoring typed-ast: markers 'python_version == "3.8"' don't match your environment
Collecting coverage==5.3
  Using cached coverage-5.3-cp39-cp39-manylinux1_x86_64.whl (228 kB)
Collecting pytest-cov==2.10.1
  Using cached pytest_cov-2.10.1-py2.py3-none-any.whl (19 kB)
Collecting pytest-mock==3.3.1
  Using cached pytest_mock-3.3.1-py3-none-any.whl (11 kB)
Collecting pytest==6.1.1
  Using cached pytest-6.1.1-py3-none-any.whl (272 kB)
Collecting toml==0.10.1
  Using cached toml-0.10.1-py2.py3-none-any.whl (19 kB)
Requirement already satisfied: attrs>=17.4.0 in ./.nox/tests-3-9/lib/python3.9/site-packages (from pytest==6.1.1->-c /tmp/tmppd82bei6 (line 176)) (20.2.0)
Collecting pluggy==0.13.1
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting iniconfig==1.1.1
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting packaging==20.4
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting py==1.9.0
  Using cached py-1.9.0-py2.py3-none-any.whl (99 kB)
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting six
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    six from https://files.pythonhosted.org/packages/ee/ff/48bde5c0f013094d729fe4b0316ba2a24774b3ff1c52d924a8a4cb04078a/six-1.15.0-py2.py3-none-any.whl#sha256=8b74bedcbbbaca38ff6d7491d76f2b06b3592611af620f8426e82dddb04a5ced (from packaging==20.4->-c /tmp/tmppd82bei6 (line 146))
nox > Session tests-3.9 failed.

[mjt91]

hypermodern-python/noxfile.py

Lines 30 to 39 in 400386b

	with tempfile.NamedTemporaryFile() as requirements:
	session.run(
	"poetry",
	"export",
	"--dev",
	"--format=requirements.txt",
	f"--output={requirements.name}",
	external=True,
	)
	session.install(f"--constraint={requirements.name}", *args, **kwargs)

I fixed this problem by adding --without-hashes parameter to the install_with_constraints function in noxfile.

[ToddG]

I am running into this issue with python3.8 (though I do not run into this with python3.7 and have not yet tried python3.9). I submitted a bug to pytype suggesting they add explicit version pinning to their requirements.txt file:

google/pytype#731

If you add the --without-hashes then doesn't that remove the guarantee that the install_with_constraints function is providing?

Also, I created my own seed repo as I did not see that @cjolowicz had created a seed repo of his own. I'll look at that and then potentially deprecate mine.

[allesmi]

I have the same problem with python 3.8 and 3.9. Only nox sessions with python 3.7 do not fail with variations of the above console output. Other packages that are missing hashes in the output of poetry export are attrs, pyYAML, and typed-ast, and probably others.

Is this behavior caused by poetry or pip or pyenv or python?

While adding the argument --without-hashes to the export commands helps to avoid the error this workaround of ignoring hashes is not the right way to go. Is the only permanent solution to nudge package maintainers to pin versions explicitly like @ToddG did for pytype? Both approaches do not appear very hypermodern to me.

[cjolowicz]

Could you try downgrading Poetry to 1.1.2, or if that doesn't help, 1.0.10?

[cjolowicz]

In the meantime, please refrain from asking maintainers of other projects to pin their requirements. This is based on a misunderstanding: Even if pytype pinned their requirements.txt, this would not affect you, because those pins would not end up in the package metadata. The pytype dependencies you see are what pytype declares in their setup.cfg. And you would not want those to be pinned, otherwise chances would be very slim that you could add several development dependencies alongside each other.

[allesmi]

After downgrading poetry to 1.1.2 the nox sessions for lint, mypy, and tests under python 3.8 run successfully.

Edit: The above issue also occurs for poetry 1.1.3

[wanderrful]

After downgrading poetry to 1.1.2 the nox sessions for lint, mypy, and tests under python 3.8 run successfully.

Edit: The above issue also occurs for poetry 1.1.3

Sadly, this did not work for me. Error message is effectively the same, though for all Nox sessions. Tried using Poetry 1.0.10, 1.1.2, and 1.1.4 (latest at time of writing). Also tried using different combinations of recent Pip versions as well as the most recent one.

[ToddG]

@wanderrful

Here is what I am currently using:

def install_with_constraints(session: Session, *args: str, **kwargs: Any) -> None:
    """Install packages constrained by Poetry's lock file.

    This function is a wrapper for nox.sessions.Session.install. It
    invokes pip to install packages inside of the session's virtualenv.
    Additionally, pip is passed a constraints file generated from
    Poetry's lock file, to ensure that the packages are pinned to the
    versions specified in poetry.lock. This allows you to manage the
    packages as Poetry development dependencies.

    Arguments:
        session: The Session object.
        args: Command-line arguments for pip.
        kwargs: Additional keyword arguments for Session.install.
    """
    with tempfile.NamedTemporaryFile() as requirements:
        session.run(
            "poetry",
            "export",
            "--dev",
            "--format=requirements.txt",
            "--without-hashes",
            f"--output={requirements.name}",
            external=True,
        )
        session.install(f"--constraint={requirements.name}", *args, **kwargs)

Then using it within nox sessions as:

@nox.session(python=_versions)
def lint(session: Session) -> None:
    """Run the code linters."""
    args = session.posargs or locations
    install_with_constraints(
        session,
        "darglint",
        "flake8",
        "flake8-annotations",
        "flake8-black",
        "flake8-docstrings",
        "flake8-isort",
        "flake8-rst-docstrings",
        "flake8_sphinx_links",
    )
    session.run("flake8", *args)

I hope that helps.

[wanderrful]

Yeah I have the same setup pretty much. I've checked the exported requirements.txt files for the dependencies it complains about and it looks fine to me. So, it makes me think that PIP itself is somehow wrongly parsing the exported requirements.txt file, which would be Poetry's fault... but if it's Poetry's fault then you'd think that using an earlier version in CICD builds would fix the problem. However, again, Poetry's different versions don't fix the issue.

It's pretty strange and makes me wonder if the DockerHub python slim images I'm using are themselves somehow the problem... but they haven't been updated for 7 months, so maybe it's not at a lower layer?

[wanderrful]

Update- I got it!

I was on the right track, with my previous comment above. However, instead of looking down to the Docker image layer, I had forgotten that there is one last layer before we get to Docker, which is virtualenv!

Manually pinning virtualenv to version 20.0.26 (current version is 20.2.2) fixes the issue. Not sure if it's just this specific current version that introduces the issue, though.

[cjolowicz]

Glad you found a workaround. I would suspect that this is triggered by changes to constraints file handling in pip's new resolver. Pip is bundled with virtualenv. You could then also "solve" this by pinning pip to a version before the new resolver became the default, setting the environment variable VIRTUALENV_PIP to 20.2.4

[shadycuz]

We also were having this issue ^.

Should an issue be raised with pip @cjolowicz or is this not a bug, just something we are doing wrong?

It seems like it's a bug if the error is the hashes are not present, but they are.

Edit: based on the discussion pypa/pip#8792 its a design decision and not a bug.

[wanderrful]

@cjolowicz is right that just pinning Pip to 20.2.4, which is the version right before the current one at time of writing, will have the same effect as the solution I talked about above.

So we're just doing pip install pip==20.2.4 for now on our end to get by in CICD builds.

[shadycuz]

@wanderrful That didn't work for me =/ https://github.com/DontShaveTheYak/cloud-radar/actions/runs/420486474

Unless you mean pinning pip with poetry or some other way? Pinning virtualenv worked. I did not try the environment variable approach.

I ended up just using --without-hashes as pinning the version was not a long-term fix since it's not a bug but the behavior going forward.

[wanderrful]

For me, doing the environment variable approach recommended above and then also pinning the Pip version itself together worked out fine.

One difference between your build and mine is that you're using the latest version of Nox, whereas we've pinned the previous version of it (2020.5.24). Nox has a virtualenv as a dependency, so I wonder if that might somehow be getting in the way.

Also, we're using poetry==1.0.9 as a pinned install rather than the latest version. Not sure where the issue is coming from for you but my guess is probably that it's coming from deeper in the dependency tree with the version numbers involved of everything else.

[Harrison88]

The cause definitely seems to be the changes to pip with the new resolver becoming the default in version 20.3. Looks like the way constraints file work has changed in this version. pypa/pip#9020 and https://pip.pypa.io/en/latest/user_guide/#watch-out-for cleared it up for me.

It seems that the only solution, other than not upgrading pip, is to add "--without-hashes" to the "poetry export" command in install_with_constraints. I couldn't find anything else, anyway.

[mjt91]

Not sure if it helps, but I can only second this what @Harrison88 said. For me with a fresh install and newest cookiecutter template this is giving me errors.

I think this is closely connected to this issue regarding virtualenv and the new pip resolver here. This got reported in the anaconda environment, but still holds true for me, since I have both - miniconda and python versions 3.9 and 3.8 installed via pyenv.

nox > Running session mypy-3.8
nox > Creating virtual environment (virtualenv) using python.exe in .nox\mypy-3-8
nox > Command $HOME\.local\pipx\venvs\nox\scripts\python.exe -m virtualenv $HOME\mjt\overwatch\.nox\mypy-3-8 -p $HOME\Miniconda3\python.exe failed with exit code 1:
FileNotFoundError: [Errno 2] No such file or directory: '$HOME\\Miniconda3\\Lib\\venv\\scripts\\nt\\python.exe'
nox > Session mypy-3.8 failed.

Here is my poetry config --list output:

cache-dir = "$HOME\\AppData\\Local\\pypoetry\\Cache"
experimental.new-installer = true
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.path = "{cache-dir}\\virtualenvs"  # $HOME\AppData\Local\pypoetry\Cache\virtualenvs

for Poetry version 1.1.4 and 1.1.2

[staticdev]

After trying many combinations of pip, virtualenv, poetry and nox-poetry the combination that worked best for me with latest versions is:

pip==20.3.3
nox==2020.12.31
nox-poetry==0.7.0
poetry==1.0.10
virtualenv==20.3.1

Basically, new poetry not working well with new virtualenv (and vice-versa). pip version does not seem to impact on hashes problem.

[joclement]

For other people coming along here: An alternative solution to downgrading pip to 20.2.4 might be to use the old resolver in pip 20.3 (check here for more info) by setting it's use in an environment variable or the config, on Linux for the config this would be in ~/.config/pip/pip.conf:

[global]
use-deprecated=legacy-resolver

This however has no influence on the virtualenv pip AFAIK.

[staticdev]

Update from my last comment. Now using all latest versions, everything seems to be working together perfectly:

pip==20.3.3
nox==2020.12.31
nox-poetry==0.7.1
poetry==1.1.4
virtualenv==20.4.0

@cjolowicz any chance we get a new release of cookiecutter with these updated versions?

[cjolowicz]

hi @staticdev

@cjolowicz any chance we get a new release of cookiecutter with these updated versions?

2021.1.29 was just released.

[cjolowicz]

For everyone not using the Cookiecutter and/or nox-poetry, I would recommend passing --without-hashes to poetry export, and using pip's new resolver.

[flickerfly]

This seems to continue to persist in 3.10.

$ nox -rs mypy
nox > Running session mypy
nox > Creating virtual environment (virtualenv) using python.BAT in .nox\mypy
nox > poetry export --dev --format=requirements.txt '--output=C:\Users\RITCHI~1.JOS\AppData\Local\Temp\tmpxcsboakw'
nox > python -m pip install '--constraint=C:\Users\RITCHI~1.JOS\AppData\Local\Temp\tmpxcsboakw' mypy
nox > Command python -m pip install '--constraint=C:\Users\RITCHI~1.JOS\AppData\Local\Temp\tmpxcsboakw' mypy failed with exit code 1:
Collecting mypy
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    mypy from https://files.pythonhosted.org/packages/37/bf/5729b25e84b0048f4ca249a10f5d5c1cc9210b40a7496435b4b872b874b6/mypy-0.931-cp310-cp310-win_amd64.whl#sha256=7b3f6f557ba4afc7f2ce6d3215d5db279bcf120b3cfd0add20a5d4f4abdae5bc
nox > Session mypy failed.