Merge pull request #337 from gkreitz/release_fix

Prepare for release 1.20250605 and for moving development from `devel` to `master`

Author: gkreitz

LICENSE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2010-2019 Kattis and all respective contributors
+Copyright (c) Kattis and all respective contributors
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -1,12 +1,12 @@
 # Kattis Problem Tools
 
-master:
-![Master Build Status](https://github.com/kattis/problemtools/actions/workflows/python-app.yml/badge.svg?branch=master)
-develop:
-![Develop Build Status](https://github.com/kattis/problemtools/actions/workflows/python-app.yml/badge.svg?branch=develop)
+![Build Status](https://github.com/kattis/problemtools/actions/workflows/python-app.yml/badge.svg?branch=master)
+[![pre-commit](https://img.shields.io/badge/pre--commit-enabled-brightgreen?logo=pre-commit)](https://github.com/pre-commit/pre-commit)
 
-These are tools to manage problem packages using the Kattis problem package
-format.
+These are tools to manage problem packages using the Kattis [problem package
+format](https://www.kattis.com/problem-package-format/). The problem package
+specification is developed in the [problem package format
+repository](https://github.com/Kattis/problem-package-format).
 
 
 ## Programs Provided
@@ -47,11 +47,15 @@ A few examples of problem packages can be found in [examples](examples).
 There are four supported ways of installing and running problemtools.
 (For non-Linux users, "Method 2" below, to use Docker, is probably the least painful.)
 
+Note that in all methods except for "Method 2", you must manually install
+dependencies such as LaTeX and tools for any languages you want to use. See
+[Requirements and compatbility](#requirements-and-compatibility) for details.
+
 ### Method 1: Install the Python package using pipx
 
 Run
 ```
-pipx install git+https://github.com/kattis/problemtools
+pipx install problemtools
 ```
 
 In order to get the command line scripts, you need to make sure that the local
@@ -77,19 +81,30 @@ We maintain three official problemtools Docker images on Docker Hub:
 
 - [`problemtools/minimal`](https://hub.docker.com/r/problemtools/minimal/): this image only contains problemtools, no additional programming languages.  As such, it is not particularly useful on its own, but if you are organizing a contest and want to set up a problemtools environment containing exactly the right set of compilers/interpreters for your contest, this is the recommended starting point.
 
-For example, suppose you want to use the `problemtools/icpc` image.  To get started, install the [Docker CLI](https://docs.docker.com/install), and then pull the image:
+For example, suppose you want to use the `problemtools/icpc` image.  To get started (or update to the latest release), install the [Docker CLI](https://docs.docker.com/install), and then pull the image:
 
     docker pull problemtools/icpc
 
-Once the image has finished downloading, you can check that it exists on your system using `docker images`. To launch an interactive container and play around with *verifyproblem*, *problem2pdf*, and *problem2html* run:
+The most convenient way to use the container is by creating shell script(s) similar to this and add it to your `$PATH`. If you call the script `verifyproblem.sh`, you could then `verifyproblem.sh examples/hello` to use the icpc docker image to verify examples/hello:
+```sh
+#!/bin/bash
+
+if [ $1 -a -d $1 ]; then
+    docker run --rm -t -v $(dirname $(readlink -f $1)):/work problemtools/icpc verifyproblem /work/$(basename $1)
+else
+    echo No such directory: $1
+fi
+```
+
+To instead launch an interactive container and play around with *verifyproblem*, *problem2pdf*, and *problem2html* run:
 
     docker run --rm -it problemtools/icpc
 
 By default, docker containers do _NOT_ persist storage between runs, so any files you create or modify will be lost when the container stops running.  Two common ways of dealing with this are:
 
-1) Use a [bind mount](https://docs.docker.com/storage/bind-mounts/) to mount a directory on your machine into the docker container.  This can be done as follows (see Docker documentation for further details):
+1) Use a [bind mount](https://docs.docker.com/storage/bind-mounts/) to mount a directory on your machine into the docker container.  Mounting the current directory to /kattis_work_dir can be done as follows (see Docker documentation for further details):
     ```
-    docker run --rm -it -v ${FULL_PATH_TO_MOUNT}:/kattis_work_dir problemtools/icpc
+    docker run --rm -it -v $(pwd):/kattis_work_dir problemtools/icpc
     ```
 2) Persist any changes you want to keep to a remote file system/source control (e.g., a remote Git repository; note, however, that you would first need to install Git in the image).
 
@@ -211,7 +226,8 @@ problemtools' configuration:
 ## Requirements and compatibility
 
 To build and run the tools, you need Python 3 with the YAML and PlasTeX libraries,
-and a LaTeX installation.
+and a LaTeX installation. You must also install language tools (e.g., compilers)
+for any languages used in problem packages.
 
 ### Ubuntu
 

admin/build_pypi_packages.sh

@@ -4,6 +4,10 @@ set -e
 ALLOW_DIRTY=false
 TAG=develop
 
+echo N.B., this script is solely to allow for local testing of whl/src.
+echo To actually build and push things to pypi, trigger the pypi flow
+echo in github actions, https://github.com/Kattis/problemtools/actions
+
 while getopts "d" opt; do
     case $opt in
         d) ALLOW_DIRTY=true ;;

admin/make_release.sh

@@ -1,10 +1,17 @@
 #!/usr/bin/env bash
 #
-# Uses git flow and gbp tools (available through Ubuntu packages
-# git-flow, git-buildpackage)
+# Uses gbp (available through Ubuntu package git-buildpackage)
 
 set -e
 
+ALLOW_DIRTY=false
+while getopts "d" opt; do
+    case $opt in
+        d) ALLOW_DIRTY=true ;;
+        \?) echo "Invalid option: -$opt" ;;
+    esac
+done
+
 ROOT=$(readlink -f $(dirname $0)/..)
 VERSION=1.$(date +%Y%m%d)
 
@@ -16,20 +23,50 @@ if [ "$(git tag -l v$VERSION)" != "" ]; then
     VERSION=$VERSION-rev$REV
 fi
 
-set -x
 
-git flow release start --showcommands $VERSION
+# Steps:
+#   Pick a version (done by the above loop)
+#   Update debian/changelog using gbp
+#   Create and merge pull request with the updated debian/changelog
+#   Create a github release (using web UI) or gh
+#   Push to pypi using github action
+#   Push to docker using admin/update_docker.sh v$VERSION
+
+CHANGELOG_VERSION=$(dpkg-parsechangelog -l $ROOT/debian/changelog --show-field Version)
+if [[ $CHANGELOG_VERSION == $VERSION ]]; then
+    echo "Debian changelog seems updated"
+else
+    echo "Updating debian changelog (this is surprisingly slow)"
+    EMAIL=$(git config user.email) gbp dch $ROOT --release --new-version=$VERSION --ignore-branch --git-author --debian-tag='v%(version)s' --debian-branch=release/$VERSION --spawn-editor=never
+    echo "Please commit the updated changelog, do a pull request, and get it merged, then run this script again on an up-to-date master branch"
+    exit 0
+fi
 
-# Update _version.py
-$ROOT/admin/update_version.py.sh $VERSION
+cd $(dirname $(readlink -f $0))
 
-# Update debian/changelog
-gbp dch $ROOT --release --new-version=$VERSION --git-author --debian-tag='v%(version)s' --debian-branch=release/$VERSION --spawn-editor=never
+if [[ -n $(git status -s) ]]; then
+    echo "Repository is dirty."
+    git status -s
+    [[ "${ALLOW_DIRTY}" != "true" ]] && exit 1
+fi
 
-git add $ROOT/problemtools/_version.py $ROOT/debian/changelog
-git commit -m "Release of version $VERSION: bump version in problemtools/_version.py and debian/changelog"
+GITTAG=master
+if [[ $(git rev-parse --abbrev-ref HEAD) != ${GITTAG} && $(git describe --exact-match --tags 2>/dev/null) != ${GITTAG} ]]; then
+    echo "Repository is currently not on branch/tag ${GITTAG}."
+    [[ "${ALLOW_DIRTY}" != "true" ]] && exit 1
+fi
+
+THIS_REPO_VERSION=$(git -C $(dirname -- "$0") rev-parse HEAD)
+UPSTREAM_VERSION=$(git -C $(dirname -- "$0") ls-remote upstream master | cut -f1)
+if [[ $THIS_REPO_VERSION != $UPSTREAM_VERSION ]]; then
+    echo "Warning: git head of repo does not match upstream. You likely want to update this repo"
+    [[ "${ALLOW_DIRTY}" != "true" ]] && exit 1
+fi
 
-git flow release finish --showcommands --message "Release $VERSION" $VERSION
 
-echo "After pushing changes to GitHub, please run"
-echo "  $ROOT/admin/update_docker.sh v$VERSION"
+echo "Below is untested, echoing commands instead of running them"
+echo "Creating a draft release on github"
+echo gh -R Kattis/problemtools release create -d v$VERSION
+echo "After finalizing the release on GitHub, please:"
+echo " - trigger the pypi release workflow"
+echo " - run $ROOT/admin/update_docker.sh v$VERSION"

admin/update_docker.sh

@@ -2,7 +2,8 @@
 set -e
 
 ALLOW_DIRTY=false
-TAG=develop
+GITTAG=master
+DOCKERTAG=develop
 UPDATE_LATEST=false
 
 while getopts "d" opt; do
@@ -15,7 +16,8 @@ done
 shift $((OPTIND-1))
 
 if [ "$1" != "" ]; then
-    TAG=$1
+    GITTAG=$1
+    DOCKERTAG=$1
     UPDATE_LATEST=true
 fi
 
@@ -27,19 +29,21 @@ if [[ -n $(git status -s) ]]; then
     [[ "${ALLOW_DIRTY}" != "true" ]] && exit 1
 fi
 
-if [[ $(git rev-parse --abbrev-ref HEAD) != ${TAG} && $(git describe --exact-match --tags 2>/dev/null) != ${TAG} ]]; then
-    echo "Repository is currently not on branch/tag ${TAG}."
+if [[ $(git rev-parse --abbrev-ref HEAD) != ${GITTAG} && $(git describe --exact-match --tags 2>/dev/null) != ${GITTAG} ]]; then
+    echo "Repository is currently not on branch/tag ${GITTAG}."
     [[ "${ALLOW_DIRTY}" != "true" ]] && exit 1
 fi
 
+echo "Updating Ubuntu base image"
+docker pull ubuntu:24.04
 
 # Make our internal images, and our githubci image. Order is important, images depend on each other
 echo "Building intermediate images, plus githubci image"
 for IMAGE in runreqs build icpclangs fulllangs githubci; do
     docker build \
         -f Dockerfile.${IMAGE} \
-        -t problemtools/${IMAGE}:${TAG} \
-        --build-arg PROBLEMTOOLS_VERSION="${TAG}" \
+        -t problemtools/${IMAGE}:${DOCKERTAG} \
+        --build-arg PROBLEMTOOLS_VERSION="${DOCKERTAG}" \
         .
 done
 
@@ -48,21 +52,21 @@ echo "Building deb"
 mkdir -p artifacts
 sudo rm -rf artifacts/deb
 # Use our build image to build a deb
-docker run --rm -v "$(pwd)/../..:/problemtools" -v "$(pwd)/artifacts/deb:/artifacts" problemtools/build:${TAG} \
+docker run --rm -v "$(pwd)/../..:/problemtools" -v "$(pwd)/artifacts/deb:/artifacts" problemtools/build:${DOCKERTAG} \
     /bin/bash -c "
         set -e ;
         mkdir /build ;
         cd /build ;
         git config --global --add safe.directory /problemtools/.git ;
-        git clone --branch ${TAG} /problemtools ;
+        git clone --branch ${GITTAG} /problemtools ;
         cd problemtools ;
         make builddeb ;
         cp ../*.deb /artifacts"
 sudo chown -R $USER:$USER artifacts/
 
 
 echo "Testing deb"
-if ! docker run --rm -t -v "$(pwd)/../..:/problemtools" -v "$(pwd)/artifacts/deb:/artifacts" problemtools/fulllangs:${TAG} \
+if ! docker run --rm -t -v "$(pwd)/../..:/problemtools" -v "$(pwd)/artifacts/deb:/artifacts" problemtools/fulllangs:${DOCKERTAG} \
     /bin/bash -c '
         set -e ;
         shopt -s extglob ;
@@ -78,17 +82,17 @@ echo "Building complete images with problemtools baked in"
 for IMAGE in minimal icpc full; do
     docker build \
         -f Dockerfile.${IMAGE} \
-        -t problemtools/${IMAGE}:${TAG} \
-        --build-arg PROBLEMTOOLS_VERSION="${TAG}" \
+        -t problemtools/${IMAGE}:${DOCKERTAG} \
+        --build-arg PROBLEMTOOLS_VERSION="${DOCKERTAG}" \
         .
 done
 
 
 if [ "${UPDATE_LATEST}" = "true" ]; then
     echo "Build complete. If you are happy with the images, run the following:"
     for IMAGE in minimal icpc full githubci; do
-        echo "    docker tag problemtools/${IMAGE}:${TAG} problemtools/${IMAGE}:latest"
-        echo "    docker push problemtools/${IMAGE}:${TAG}"
+        echo "    docker tag problemtools/${IMAGE}:${DOCKERTAG} problemtools/${IMAGE}:latest"
+        echo "    docker push problemtools/${IMAGE}:${DOCKERTAG}"
         echo "    docker push problemtools/${IMAGE}:latest"
     done
 fi