twine: command not found

[james-rms]

Filing separately from #112 , since the $PATH issues outlined there seem separate from what I'm experiencing.

Version: e71a4a4

Example failure:
https://github.com/foxglove/mcap/actions/runs/3634377846/jobs/6132519798#step:11:20

with:
    user: __token__
    password: ***
    packages_dir: python/mcap/dist
    repository_url: https://test.pypi.org/legacy/
    skip_existing: true
    verify_metadata: true
    verbose: false
    print_hash: false
  env:
    pythonLocation: /opt/hostedtoolcache/Python/[3](https://github.com/foxglove/mcap/actions/runs/3634377846/jobs/6132519798#step:11:3).7.15/x6[4](https://github.com/foxglove/mcap/actions/runs/3634377846/jobs/6132519798#step:11:4)
    PKG_CONFIG_PATH: /opt/hostedtoolcache/Python/3.7.1[5](https://github.com/foxglove/mcap/actions/runs/3634377846/jobs/6132519798#step:11:5)/x[6](https://github.com/foxglove/mcap/actions/runs/3634377846/jobs/6132519798#step:11:6)4/lib/pkgconfig
    Python_ROOT_DIR: /opt/hostedtoolcache/Python/3.7.15/x64
    Python2_ROOT_DIR: /opt/hostedtoolcache/Python/3.7.15/x64
    Python3_ROOT_DIR: /opt/hostedtoolcache/Python/3.7.15/x64
    LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.7.15/x64/lib
/usr/bin/docker run --name ac8f0bed99144e79bbebae175f60826_b0a163 --label 290506 --workdir /github/workspace --rm -e "pythonLocation" -e "PKG_CONFIG_PATH" -e "Python_ROOT_DIR" -e "Python2_ROOT_DIR" -e "Python3_ROOT_DIR" -e "LD_LIBRARY_PATH" -e "INPUT_USER" -e "INPUT_PASSWORD" -e "INPUT_PACKAGES_DIR" -e "INPUT_REPOSITORY_URL" -e "INPUT_SKIP_EXISTING" -e "INPUT_VERIFY_METADATA" -e "INPUT_VERBOSE" -e "INPUT_PRINT_HASH" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/mcap/mcap":"/github/workspace" 290506:6ac8f0bed99144e79bbebae175f60826  "__token__" "***" "https://test.pypi.org/legacy/" "python/mcap/dist" "true" "true" "false" "false"
/app/twine-upload.sh: line 50: twine: command not found

Action configuration:
https://github.com/foxglove/mcap/blob/0b5fcf170aa8cb265a38fa2ffc03cb9b53fc131c/.github/workflows/ci.yml#L256:L278

  python:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: python
    steps:
      - uses: actions/checkout@v3
        with:
          lfs: true
      - uses: actions/setup-python@v4
        with:
          python-version: 3.7
      - uses: satackey/action-docker-layer-caching@v0.0.11
        continue-on-error: true
      - run: pip install pipenv==2022.7.24
      - run: make lint
      - run: make test
      - run: make examples
      - run: make build
      - name: Publish mcap to TestPyPI
        uses: pypa/gh-action-pypi-publish@release/v1
        if: ${{ !github.event.pull_request.head.repo.fork && github.actor != 'dependabot[bot]' }}
        with:
          user: __token__
          password: ${{ secrets.TESTPYPI_API_TOKEN }}
          packages_dir: python/mcap/dist
          repository_url: https://test.pypi.org/legacy/
          skip_existing: true

[webknjaz]

That is super weird. The smoke test that we now have in GHA shows that a broken PATH does not influence this. I wonder if you could try to shed some light by adding

env:
   DEBUG: true

to the corresponding step.

It could also be useful to verify that this does not happen if the action is invoked in a dedicated job that only downloads the artifacts before calling it.

[woodruffw]

I'm also seeing this error:

Run pypa/gh-action-pypi-publish@v1.6.3
/usr/bin/docker run --name def1238cfe3e4753b6b01329ccdbdf6c_b41197 --label 290506 --workdir /github/workspace --rm -e "pythonLocation" -e "PKG_CONFIG_PATH" -e "Python_ROOT_DIR" -e "Python2_ROOT_DIR" -e "Python3_ROOT_DIR" -e "LD_LIBRARY_PATH" -e "INPUT_USER" -e "INPUT_PASSWORD" -e "INPUT_REPOSITORY_URL" -e "INPUT_PACKAGES_DIR" -e "INPUT_VERIFY_METADATA" -e "INPUT_SKIP_EXISTING" -e "INPUT_VERBOSE" -e "INPUT_PRINT_HASH" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_ID_TOKEN_REQUEST_URL" -e "ACTIONS_ID_TOKEN_REQUEST_TOKEN" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/abi3info/abi3info":"/github/workspace" 290506:def1238cfe3e4753b6b01329ccdbdf6c  "__token__" "***" "" "dist" "true" "false" "false" "false"
/app/twine-upload.sh: line 50: twine: command not found

Logs: https://github.com/woodruffw/abi3info/actions/runs/3634987726/jobs/6133656272

[suizhihao]

/app/twine-upload.sh: line 50: twine: command not found

[suizhihao]

https://github.com/opendatalab/dsdl-sdk/actions/runs/3634921092/jobs/6133519712

[woodruffw]

It could also be useful to verify that this does not happen if the action is invoked in a dedicated job that only downloads the artifacts before calling it.

My action uses the artifacts from a previous step, so it shouldn't be that:

    - name: deps
      run: python -m pip install -U setuptools build wheel

    - name: build
      run: python -m build

    - name: publish
      uses: pypa/gh-action-pypi-publish@v1.6.3
      with:
        user: __token__
        password: ${{ secrets.PYPI_TOKEN }}

    - name: sign
      uses: sigstore/gh-action-sigstore-python@v0.1.0
      with:
        inputs: ./dist/*.tar.gz ./dist/*.whl
        release-signing-artifacts: true

I'll try a debug run in a moment.

[webknjaz]

The smoke test that we now have in GHA shows that a broken PATH does not influence this.

I was wrong: it does fail — https://github.com/pypa/gh-action-pypi-publish/actions/runs/3634959022.

[woodruffw]

I'll try a debug run in a moment.

Logs: https://github.com/woodruffw/abi3info/actions/runs/3635033169/jobs/6133748800

In particular:

Run pypa/gh-action-pypi-publish@v1.6.3
/usr/bin/docker run --name a356773daf846d783af59338dc33a74_ef732c --label 290506 --workdir /github/workspace --rm -e "pythonLocation" -e "PKG_CONFIG_PATH" -e "Python_ROOT_DIR" -e "Python2_ROOT_DIR" -e "Python3_ROOT_DIR" -e "LD_LIBRARY_PATH" -e "DEBUG" -e "INPUT_USER" -e "INPUT_PASSWORD" -e "INPUT_REPOSITORY_URL" -e "INPUT_PACKAGES_DIR" -e "INPUT_VERIFY_METADATA" -e "INPUT_SKIP_EXISTING" -e "INPUT_VERBOSE" -e "INPUT_PRINT_HASH" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_ID_TOKEN_REQUEST_URL" -e "ACTIONS_ID_TOKEN_REQUEST_TOKEN" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/abi3info/abi3info":"/github/workspace" 290506:6a356773daf846d783af59338dc33a74  "__token__" "***" "" "dist" "true" "false" "false" "false"
+ set -Eeuo pipefail
+ export PATH=/usr/bin:/root/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/usr/bin:/root/.local/bin:/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ . /etc/profile
+++ id -u
++ '[' 0 -eq 0 ']'
++ PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH
++ '[' '' ']'
++ '[' -d /etc/profile.d ']'
++ for i in /etc/profile.d/*.sh
++ '[' -r '/etc/profile.d/*.sh' ']'
++ unset i
++ python -m site --user-base
+ export PATH=/github/home/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/github/home/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ python -m site --user-site
+ export PYTHONPATH=/github/home/.local/lib/python3.11/site-packages:/root/.local/lib/python3.11/site-packages
+ PYTHONPATH=/github/home/.local/lib/python3.11/site-packages:/root/.local/lib/python3.11/site-packages
+ [[ __token__ == \_\_\t\o\k\e\n\_\_ ]]
+ [[ ! *** =~ ^pypi- ]]
+ ls -A dist/abi3info-2022.12.6.post1.tar.gz
+ [[ true != \f\a\l\s\e ]]
+ twine check dist/abi3info-2022.12.6.post1-py3-none-any.whl dist/abi3info-2022.12.6.post1.tar.gz
/app/twine-upload.sh: line 50: twine: command not found

[webknjaz]

🤦‍♂️ looks like it's the same problem but with $HOME (python -m site ... invocations probably get the var passed down from the runner).

[webknjaz]

It's reproducible locally via podman run --rm -v "$(pwd):/app" -e INPUT_REPOSITORY_URL= -e INPUT_PASSWORD= -e INPUT_PRINT_HASH= -e INPUT_VERBOSE= -e INPUT_SKIP_EXISTING= -e INPUT_USER= -e INPUT_PACKAGES_DIR=dist/ -e INPUT_VERIFY_METADATA= -e PATH=utter-nonsense -e DEBUG=true -e HOME= localhost/gh-action-pypi-publish:latest

[webknjaz]

Alright... the CI passes with the hotfix: https://github.com/pypa/gh-action-pypi-publish/actions/runs/3635075803.
I'll plan to replace the user-global installs with a properly managed venv next time I'll get to the feature work here. That should improve the situation in general.