uv venv and github actions scripts / workflows

[strickvl]

Having trouble converting our pre-existing CI workflows / scripts to use uv, as you can see here. Problem relates to the virtual environments which don't seem to be retained between workflow steps.

This is maybe the easiest illustration of that:

  steps:
      - name: Checkout code
        uses: actions/checkout@v4.1.1
      - name: Set up Python
        uses: actions/setup-python@v5.0.0
        with:
          python-version: '3.10'
      - name: Install current package as editable
        run: |
          pip install uv
          uv venv
          uv pip install darglint
      - name: Check docstrings
        run: bash scripts/docstring.sh

One step installs it, and then the next step runs a script which in turn attempts to use the package, but it fails because the CI can't be found.

I don't see any docs yet, so wondering how best to handle this?

[zanieb]

Hi! You need to activate virtual environments so set VIRTUAL_ENV=./.venv after creating.

Related #1326

[strickvl]

Sorry maybe I'm being dumb here, but I've tried this and it still doesn't pick up the package.

steps:
      - name: Checkout code
        uses: actions/checkout@v4.1.1
      - name: Set up Python
        uses: actions/setup-python@v5.0.0
        with:
          python-version: '3.10'
      - name: Install current package as editable
        run: |
          pip install uv
          uv venv
          VIRTUAL_ENV=./.venv
          uv pip install darglint
      - name: Check docstrings
        run: bash scripts/docstring.sh
        env:
          VIRTUAL_ENV: ./.venv

I even threw in a source .venv/bin/activate after the uv venv and that also did nothing. Am I doing what you suggested, or am I doing it wrong?

[ericbn]

With GitHub Actions I think it makes more sense to install the dependencies globally in the virtual machine. uv does not support that yet. See #1374

In your “Check docstrings” step shown above you’re missing also updating the PATH environment variable to include “$VIRTUAL_ENV/bin”. Or you can call . .venv/bin/activate at the beginning of the step, which will take care of the VIRTUAL_ENV and PATH environment variables for you.

[dylanbstorey]

Had a similar issue, got this working for me with the following snippet :

 angreal-tests-linux:
    name: "angreal run-tests linux"
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          toolchain: 1.67.0
          override: true
      - uses: Swatinem/rust-cache@v1
      - uses: webfactory/ssh-agent@v0.7.0
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - uses: actions/setup-python@v4
        with:
          python-version: "3.12"
      - run: echo "VIRTUAL_ENV=${Python_ROOT_DIR}" >> $GITHUB_ENV
      - run: pip install uv
      - run: uv pip install maturin pytest
      - run: uv pip install angreal@.
      - run: cargo test -v -- --nocapture --test-threads=1
      - run: python -m pytest -svv

Its a bit of a hack, but basically you create the environment variable and set it to the root python and append it to the special github_env variable.

This basically tells uv to just use the system python instead oa virtual env, should be good to go,

[strickvl]

@dylanbstorey that's a neat trick, and helpful it works across operating systems too. But would be good for uv to support installations globally too to make this unnecessary.

[strickvl]

Ok and key here is that the - run: echo "VIRTUAL_ENV=${Python_ROOT_DIR}" >> $GITHUB_ENV step runs as a separate step. i.e. you can't just run that bash command from within a step that has some other uv commands as then it won't pick up the Env var.

[strickvl]

Was running into race conditions here or at least issues with steps / jobs running in parallel attempting to access the same environment, it seems. I switched back to using source ... everywhere as it seemed more controlled.

[adamtheturtle]

Ideally for me there would be clear instructions on good practice for using uv in GitHub Actions, including on Windows, and including with a cache using actions/cache.

[adamtheturtle]

Thank you @dylanbstorey .

In my use of your trick, I quote "$GITHUB_ENV" as per examples in https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-an-environment-variable and as advised by shellcheck.

SC2086:info:1:42: Double quote to prevent globbing and word splitting [shellcheck]

Ideally for me any documented example will pass actionlint.

[kdeldycke]

Can confirm @dylanbstorey's hack

      - run: echo "VIRTUAL_ENV=${Python_ROOT_DIR}" >> $GITHUB_ENV

is working fine in my case:

https://github.com/kdeldycke/workflows/blob/9e840d7c00ccb5a62272d437d2fe23c2cfbf866d/.github/workflows/lint.yaml#L68-L71

[AlexWaygood]

The previous solutions suggested in this thread aren't cross-platform:

• echo is a Unix-only command; you need different syntax to set an environment variable on Windows
• Even if you set the environment variable correctly on Windows, setting VIRTUAL_ENV to the Python root dir without creating a virtual environment doesn't work on Windows, due to the fact that python.exe isn't in a Scripts/ directory on Windows if you don't have a virtual environment activated, and uv isn't (yet?) aware of that.

All that means that the previous suggested solutions won't work in a workflow job that runs on multiple operating systems in CI. After much debugging with @MichaReiser, here's a workflow job that I'm now successfully using to run tests for my hobby project typeshed-stats. It runs on Linux, MacOS and Windows in CI:

jobs:
  pytest-tests:
    name: Run tests with pytest
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: ["ubuntu-latest", "windows-latest", "macos-latest"]
        python-version: ["3.10", "3.11", "3.12"]
      fail-fast: false

    steps:
      - uses: actions/checkout@v4
      - name: Set up Python ${{ matrix.python-version }} on ${{ matrix.os }}
        uses: actions/setup-python@v5
        with:
          python-version: ${{ matrix.python-version }}
          allow-prereleases: true
      - name: Create and activate a virtual environment (Windows)
        if: ${{ runner.os == 'Windows' }}
        # uv doesn't (yet) allow us to install packages globally;
        # we have to create and activate a virtual environment
        run: |
          irm https://astral.sh/uv/install.ps1 | iex
          uv venv .venv
          "VIRTUAL_ENV=.venv" | Out-File -FilePath $env:GITHUB_ENV -Append
          "$PWD/.venv/Scripts" | Out-File -FilePath $env:GITHUB_PATH -Append
      - name: Create and activate a virtual environment (Unix)
        if: ${{ runner.os != 'Windows' }}
        # uv doesn't (yet) allow us to install packages globally;
        # we have to create and activate a virtual environment
        run: |
          curl -LsSf https://astral.sh/uv/install.sh | sh
          uv venv .venv
          echo "VIRTUAL_ENV=.venv" >> $GITHUB_ENV
          echo "$PWD/.venv/bin" >> $GITHUB_PATH
      - name: Install dependencies
        run: uv pip install -e ".[pytest]"
      - run: uv pip freeze
      - name: Run tests under coverage
        run: |
          coverage run -m pytest
          coverage report

[adamtheturtle]

@AlexWaygood, @MichaReiser Thank you for investigating this.

If you want to use echo on Windows in GitHub Actions, you can set shell: bash on a GitHub Action.

[AlexWaygood]

If you want to use echo on Windows in GitHub Actions, you can set shell: bash on a GitHub Action.

@adamtheturtle we tried that but it caused other things to break ;) take a look at the test failures on AlexWaygood/typeshed-stats#191.

[strickvl]

Had / having similar issues on our CI. Choosing to wait for a more stable way for this global install to happen which I believe will be worked on next week.

[mgaitan]

The following worked here.

jobs:
  build:

    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11"]

    steps:
      - uses: actions/checkout@v3
      - name: Set up Python ${{ matrix.python-version }}
        uses: actions/setup-python@v4
        with:
          python-version: ${{ matrix.python-version }}
      
      - name: Install dependencies
        run: |
          python -m pip install uv
          uv venv
          uv pip install -e .[dev]
      
      - name: Activate virtualenv
        run: |
          . .venv/bin/activate
          echo PATH=$PATH >> $GITHUB_ENV

      - name: Test with pytest
        run: pytest

[danielhollas]

Ideally for me there would be clear instructions on good practice for using uv in GitHub Actions, including on Windows, and including with a cache using actions/cache.

I'll just note that setup-python action also has a builtin cache. @AlexWaygood have you done any testing to see if using this cache meaningfully speeds things up? Given how fast uv is, I wouldn't be surprised if there wasn't much difference. :-)

[AlexWaygood]

@AlexWaygood have you done any testing to see if using this cache meaningfully speeds things up?

Nope! But it's a good point!

Given how fast uv is, I wouldn't be surprised if there wasn't much difference. :-)

I think that would likely depend quite a lot on how complex it is to resolve the requirements set. But for simple requirements sets, I'd wager you're likely right.

[danielhollas]

on how complex it is to resolve the requirements set.

The way I understand it, the cache does not really speeds up resolution itself, no? Well, unless you do not run uv pip install at all upon cache hit.

[AlexWaygood]

The way I understand it, the cache does not really speeds up resolution itself, no? Well, unless you do not run uv pip install at all upon cache hit.

oh I don't actually really know anything about how the caching works. Should probably not have implied that I did 😆

[strickvl]

Closing this issue as the --system flag has basically solved this issue for me. Thanks uv team!

[zanieb]

The cache does improve resolution times because sometimes we need to build source distributions to determine their requirements during resolution which is generally slow — the cache contains these builds.

[zanieb]

Hi! For those of you who are subscribed to this and are using echo "VIRTUAL_ENV=${Python_ROOT_DIR}" as a work-around — as of 0.2.0 this is no longer supported and we'd recommend using the --system flag or --python <path> instead.

[elronbandel]

Can anyone share an example of how to set up uv in GitHub actions with --system/--python?

[zanieb]

@elronbandel here's an example:

• Use UV_SYSTEM_PYTHON to allow the system Python interpreter inventree/InvenTree#7317