Upgrade to pyproject.toml and pypa/gh-action-pypi-publish

[simonw]

• https://til.simonwillison.net/python/pyproject
• https://til.simonwillison.net/pypi/pypi-releases-from-github

[simonw]

Tried the in-development version of this to build datasette-test:

./LICENSE
./pyproject.toml
./tests
./tests/test_datasette_test.py
./README.md
./setup.py
./.gitignore
./datasette_test
./datasette_test/__init__.py
./.github
./.github/workflows
./.github/workflows/publish.yml
./.github/workflows/test.yml

[project]
name = "datasette-test"
version = "0.1"
description = "Utilities to help write tests for Datasette plugins and applications"
readme = "README.md"
requires-python = ">=3.8"
authors = [{name = "Simon Willison"}]
license = {text = "Apache-2.0"}
classifiers = [
    "License :: OSI Approved :: Apache Software License"
]
dependencies = [

]

[project.urls]
Homepage = "https://github.com/simonw/datasette-test"
Changelog = "https://github.com/simonw/datasette-test/releases"
Issues = "https://github.com/simonw/datasette-test/issues"
CI = "https://github.com/simonw/datasette-test/actions"


[project.optional-dependencies]
test = ["pytest"]

This worked:

pip install -e '.[test]'

The publish.yml looked like this:

name: Publish Python Package

on:
  release:
    types: [created]

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v5
      with:
        python-version: ${{ matrix.python-version }}
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install '.[test]'
    - name: Run tests
      run: |
        pytest
  deploy:
    runs-on: ubuntu-latest
    needs: [test]
    environment: release
    permissions:
      id-token: write
    steps:
    - uses: actions/checkout@v4
    - name: Set up Python
      uses: actions/setup-python@v5
      with:
        python-version: "3.12"
        cache: pip
        cache-dependency-path: pyproject.toml
    - name: Install dependencies
      run: |
        pip install setuptools wheel build
    - name: Build
      run: |
        python -m build
    - name: Publish
      uses: pypa/gh-action-pypi-publish@release/v1

Compare with this one which I know works: https://github.com/datasette/datasette-build/blob/02e612b667aaa0bc07da24c81023fa6b97d6c270/.github/workflows/publish.yml

Diff here: https://gist.github.com/simonw/968f85a24fead6bf814622b5390d7124/revisions

I think it's going to work. I'm going to risk committing this here, then test it live.

[simonw]

Annoying: https://github.com/simonw/python-lib/actions/runs/7546633197/job/20544794655

I think I can fix that by manually editing that workflow myself.

[simonw]

Yeah I need to run this manually on my own laptop to help solve this:

[simonw]

git clone git@github.com:simonw/python-lib-template-demo
git clone git@github.com:simonw/python-lib
mv python-lib-template-demo python-lib-template-demo-old
cat python-lib/input-for-demo.txt | cookiecutter python-lib
# Now python-lib-template-demo exists as well
mv python-lib-template-demo-old/.git python-lib-template-demo
cd python-lib-template-demo
git diff
# Looks good
git commit -a -m 'Updates'
git push

[simonw]

That seemed to work - here's the new example output, which passes its tests: https://github.com/simonw/python-lib-template-demo/tree/8ca44689

And here's the full diff simonw/python-lib-template-demo@f417323...8ca4468

[simonw]

Next test: use https://github.com/new?template_name=python-lib-template-repository&template_owner=simonw to create a datasette-test package and then ship it!

[simonw]

OK, that created this: https://github.com/datasette/datasette-test/tree/8d5f8262dc3a88f3c6d97f0cef3b55264cabc695

[simonw]

The test.yml workflow in that is completely wrong - full of setup.py references: https://github.com/datasette/datasette-test/blob/8d5f8262dc3a88f3c6d97f0cef3b55264cabc695/.github/workflows/test.yml

[simonw]

I'm really confused. https://github.com/simonw/python-lib/blob/main/%7B%7Bcookiecutter.hyphenated%7D%7D/.github/workflows/test.yml has the correct stuff in it. Why did datasette-test get the old version?

[simonw]

Oh! That's because I needed to update these specific files:

• https://github.com/simonw/python-lib-template-repository/blob/main/.github/workflows/test.yml
• https://github.com/simonw/python-lib-template-repository/blob/main/.github/workflows/publish.yml

That's my workaround for the problem where you can't push an update to a workflow - I include copies of them in the template repository itself.

[simonw]

Fixing that like so, after generating a fresh output in /tmp:

cd python-lib-template-repository
cp /tmp/datasette-test/.github/workflows/test.yml .github/workflows/test.yml
cp /tmp/datasette-test/.github/workflows/publish.yml .github/workflows/publish.yml 

[simonw]

I'm going to try generating a repo from that manually: https://github.com/new?template_name=python-lib-template-repository&template_owner=simonw

[simonw]

Created: https://github.com/simonw/python-lib-issue-6

I'll copy the workflows from that into https://github.com/datasette/datasette-test/

[simonw]

I'm going to ship 0.1 of datasette/datasette-test to see if the publish workflow works.

[simonw]

Trying that with a Trusted Publisher:

[simonw]

https://github.com/datasette/datasette-test/actions/runs/7547266249/job/20546831786 it worked!

https://pypi.org/project/datasette-test/0.1/ was released by the new template.

[simonw]

There's one sharp edge left: the tests fail the first time the repository is cloned from the template: simonw/python-lib-issue-6@54887b2

https://github.com/simonw/python-lib-issue-6/actions/runs/7547200692/job/20546630262 failed:

[simonw]

It doesn't like that there are no tests.

[simonw]

One more test: https://github.com/simonw/python-lib-issues-6-demo-2

Tests now pass on initial commit: simonw/python-lib-issues-6-demo-2@73d216e

[simonw]

The setup.py did show some warnings: https://github.com/simonw/python-lib-issues-6-demo-2/actions/runs/7547373090

[simonw]

I'll use that to create a new test repo, check for no errors, then delete that repo.

[simonw]

Yes, that worked - I added whitespace to the tests/*.py file too to make sure the tests passed.

[simonw]

This is done!

One last thing: I'm going to update the documentation on https://github.com/simonw/python-lib to remind me how to update the template in the future.

[simonw]

That documentation is here: https://github.com/simonw/python-lib/blob/main/README.md#notes-on-updating-this-cookiecutter-template