{env_name} substitution broken since version 4.14.1

[rhabarberbarbara]

Issue

I'm creating a pipeline where a coverage report is generated over multiple test runs (spanning different Python versions). To do this, I want to include the tox environment in the coverage file name, so the artifacts don't overwrite each other in the pipeline job that combines the coverage files.

It's configured as follows (with some sections left out):

[testenv]
package = wheel
deps =
    coverage[toml] ~= 7.0
commands_pre =
    python -m coverage erase
commands =
    python -m coverage run --data-file=.coverage.{env_name} --branch --source my.project -m unittest discover {posargs:tests/unit}

This worked fine up to tox 4.14.0:

py311: commands_pre[0]> python -m coverage erase
py311: commands[0]> python -m coverage run --data-file=.coverage.py311 --branch --source my.project -m unittest discover tests/unit

However, from tox 14.4.1 onwards, the env_name variable isn't substituted anymore, and just comes through as {env_name}

py311: commands_pre[0]> python -m coverage erase
py311: commands[0]> python -m coverage run --data-file=.coverage.{env_name} --branch --source myproject -m unittest discover tests/unit

Environment

Provide at least:

• OS: Windows 10

Output of pip list of the host Python, where tox is installed

Package       Version
------------- -------
cachetools    5.3.3
chardet       5.2.0
colorama      0.4.6
distlib       0.3.8
filelock      3.14.0
packaging     24.1
pip           24.0
platformdirs  4.2.2
pluggy        1.5.0
pyproject-api 1.6.1
setuptools    65.5.0
tomli         2.0.1
tox           4.14.1
virtualenv    20.26.2

Output of running tox

See above.

Minimal example

I tried to create a minimal example by creating a tox configuration that doesn't build and install a package, but for some reason the substitution in the command did work there. So somehow the package building and installing is important for the issue to occur.
Unfortunately, I can't share my entire tox config, as it's part of a corporate repo.

[shekhuverma]

I tried to re-create this but in my case, I'm able to use {env_name}.
I used the following code for testing and I'm getting the name "temp" printed out on the screen.

[testenv:temp]
commands =
    python -c "print(r'{env_name}')"

[rhabarberbarbara]

Hi @shekhuverma,

Indeed, that works for me as well. But as noted earlier:

So somehow the package building and installing is important for the issue to occur.

I did some further investigation, and managed to create a minimal reproducible example:
https://github.com/rhabarberbarbara/tox-issue-3292

This example shows that the issue did not exist in tox 4.14.0, while it does in 4.14.1:

MINGW64 ~/github/tox-issue-3292 (main)
$ py -3.11 -m tox run
.pkg: install_requires> python -I -m pip install setuptools>61 setuptools_scm[toml]>=7.0
.pkg: _optional_hooks> python C:\Python\311\Lib\site-packages\pyproject_api\_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python C:\Python\311\Lib\site-packages\pyproject_api\_backend.py True setuptools.build_meta __legacy__
.pkg: install_requires_for_build_wheel> python -I -m pip install wheel
.pkg: build_wheel> python C:\Python\311\Lib\site-packages\pyproject_api\_backend.py True setuptools.build_meta __legacy__
py311: install_package> python -I -m pip install --force-reinstall --no-deps C:\Users\rhabarberbarbara\github\tox-issue-3292\.tox\.tmp\package\1\mypackage-0.1.dev2+g833b951-py3-none-any.whl
py311: commands[0]> python -m mypackage run --data-file=.coverage.{env_name} --branch
Running mypackage with: ['run', '--data-file=.coverage.{env_name}', '--branch']
  py311: OK (23.69=setup[23.53]+cmd[0.16] seconds)
  congratulations :) (24.03 seconds)

MINGW64 ~/github/tox-issue-3292 (main)
$ py -3.11 -m tox run -x 'tox.requires=tox<4.14.1'
ROOT: will run in automatically provisioned tox, host C:\Python\311\python.exe is missing [requires (has)]: tox<4.14.1 (4.15.1)
ROOT: install_deps> python -I -m pip install tox<4.14.1 tox>=4
ROOT: provision> .tox\.tox\Scripts\python.exe -m tox run -x tox.requires=tox<4.14.1
.pkg: _optional_hooks> python C:\Users\rhabarberbarbara\github\tox-issue-3292\.tox\.tox\Lib\site-packages\pyproject_api\_backend.py True setuptools.build_meta __legacy__
.pkg: get_requires_for_build_wheel> python C:\Users\rhabarberbarbara\github\tox-issue-3292\.tox\.tox\Lib\site-packages\pyproject_api\_backend.py True setuptools.build_meta __legacy__
.pkg: build_wheel> python C:\Users\rhabarberbarbara\github\tox-issue-3292\.tox\.tox\Lib\site-packages\pyproject_api\_backend.py True setuptools.build_meta __legacy__
py311: install_package> python -I -m pip install --force-reinstall --no-deps C:\Users\rhabarberbarbara\github\tox-issue-3292\.tox\.tmp\package\2\mypackage-0.1.dev2+g833b951-py3-none-any.whl
py311: commands[0]> python -m mypackage run --data-file=.coverage.py311 --branch
Running mypackage with: ['run', '--data-file=.coverage.py311', '--branch']
.pkg: _exit> python C:\Users\rhabarberbarbara\github\tox-issue-3292\.tox\.tox\Lib\site-packages\pyproject_api\_backend.py True setuptools.build_meta __legacy__
  py311: OK (3.53=setup[3.39]+cmd[0.14] seconds)
  congratulations :) (4.02 seconds)

The issue somehow seems tied to the presence of the .build_env and/or deploy environments. But it's strange, since neither of these are used in the two tox runs above.

[shekhuverma]

Bisected to #3237

Trying to fix the issue

[webknjaz]

Oh, I was just about to post an issue about this. It was super weird to debug. removing some testsenvs made it work adding more (even empty) triggered it again. I still don't understand the underlying cause but I ended up boiling it down to a small reproducer:

# tox.ini
[testenv:.pkg]

[testenv:bug-demo]
package = skip

with this, tox crashes as follows:

$ tox r -e bug-demo -vvvvv
ROOT: 108 D setup logging to NOTSET on pid 2907185 [tox/report.py:221]
ROOT: 156 E HandledError| .pkg cannot self-package [tox/run.py:23]

Interestingly, dropping package = skip, makes it pass. Alternatively, dropping [testenv:.pkg] makes it pass. Replacing [testenv:.pkg] with [pkgenv] does not trigger the problem and it passes.

cc @gaborbernat

P.S. The [testenv:.pkg] idea is coming from @hynek's https://hynek.me/articles/turbo-charge-tox/ and https://tox.wiki/en/latest/config.html#wheel_build_env.

[gaborbernat]

PR Welcome. Note, however, that this is probably a difficult problem.

[webknjaz]

@gaborbernat yeah, I figured it's difficult having spent hours... Any hints? Could you confirm that using [pkgenv] is a valid workaround?

[gaborbernat]

I don't know what that stands for, but it is not recognized by the tool, unless you somehow configure it via references to use it.

[webknjaz]

@gaborbernat the migration guide says it's the base env for building packages, IIUC: https://tox.wiki/en/latest/upgrading.html#packaging-configuration-and-inheritance.

[gaborbernat]

Alright, forgot about that change. Then it should work.

[webknjaz]

xref possible duplicate: #3238

[gaborbernat]

Oh, I see. You are correct. This is a duplicate issue closing as such.

[webknjaz]

@gaborbernat I was just about to ask you to also label this as level:hard 😃

Anyway, I think there might be some useful context spread across these two issues so I wanted to make sure that the details get migrated over to the other one.

@rhabarberbarbara @shekhuverma could you post/move any relevant details present in your investigation over at #3238 (those that might be missing there)? This will help consolidate all the info gathered by different people in disconnected places...