Benchmarking

.github/workflows/tests.yml

@@ -11,18 +11,49 @@ jobs:
     runs-on: ubuntu-18.04
     steps:
       - name: Cancel Previous Runs
-        uses: styfle/cancel-workflow-action@0.6.0
+        uses: styfle/cancel-workflow-action@0.10.0
         with:
           access_token: ${{ github.token }}
-      - uses: actions/checkout@v2
-      - uses: actions/setup-python@v2
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
       - name: install clang-format
         if: steps.clang_format.outputs.cache-hit != 'true'
         run: |
           sudo apt-get remove -y clang-6.0 libclang-common-6.0-dev libclang1-6.0 libllvm6.0
           sudo apt-get autoremove
           sudo apt-get install clang-format clang-format-6.0
-      - uses: pre-commit/action@v2.0.0
+      - uses: pre-commit/action@v3.0.0
+
+  benchmark:
+    name: Benchmark
+    runs-on: ubuntu-latest
+    steps:
+      - name: Cancel Previous Runs
+        uses: styfle/cancel-workflow-action@0.10.0
+      - uses: actions/checkout@v3
+      - uses: actions/setup-python@v4
+        with:
+          python-version: '3.10'
+          cache: 'pip'
+          cache-dependency-path: python/requirements/development.txt
+      - name: Install deps
+        run: pip install -r python/requirements/development.txt
+      - name: Build module
+        run: |
+          cd python
+          python setup.py build_ext --inplace
+      - name: Run benchmarks
+        run: |
+          cd python/benchmark
+          pip uninstall -y tskit
+          python run.py
+      - name: Upload Results
+        uses: actions/upload-artifact@v2
+        with:
+          name: benchmark-results
+          path: python/benchmark
 
   test:
     name: Python
@@ -37,12 +68,12 @@ jobs:
         shell: bash
     steps:
       - name: Cancel Previous Runs
-        uses: styfle/cancel-workflow-action@0.6.0
+        uses: styfle/cancel-workflow-action@0.10.0
         with:
           access_token: ${{ github.token }}
 
       - name: Checkout
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Cache conda and dependencies
         id: cache
@@ -143,12 +174,12 @@ jobs:
         shell: msys2 {0}
     steps:
     - name: Cancel Previous Runs
-      uses: styfle/cancel-workflow-action@0.6.0
+      uses: styfle/cancel-workflow-action@0.10.0
       with:
         access_token: ${{ github.token }}
 
     - name: 'Checkout'
-      uses: actions/checkout@v2
+      uses: actions/checkout@v3
 
     - name: Setup MSYS2 ${{matrix.sys}}
       uses: msys2/setup-msys2@v2

docs/development.md

@@ -1076,6 +1076,45 @@ pre-commit installed at .git/hooks/pre-commit
 ```
 
 
+## Benchmarking
+
+Tskit has a simple benchmarking tool to help keep track of performance.
+
+### Running benchmarks
+
+The benchmark suite can be run with:
+
+```bash
+> cd python/benchmark
+> python run.py
+```
+
+Results are written to `bench-results.json` in the same folder. Note that if any version of `tskit`
+is installed then that will be used for the benchmarking. To use the local development version of
+tskit ensure you have `pip uninstall tskit` before running the benchmarking. The version used is
+shown in the header of the report.
+
+### Adding a new benchmark
+
+The benchmarks are specified by the `config.yaml` file in `python/benchmark`. To add a new benchmark 
+add an entry to the `benchmarks` dictionary. For example:
+
+```yaml
+  - code: do_my_thing({option_name})
+    setup: |
+      import a_module
+    name: my_benchmark #optional, the code is used by default
+    parameters:
+      option_name:
+        - "reticulate_splines"
+        - "foobar"
+```
+
+Strings are interpreted as Python f-strings, so you can use the `parameters` dictionary to provide
+values that will be interpolated into both the `setup` and `code` strings.
+
+The suite can be run for all released versions with the `run-for-all-releases.py` script. 
+
 ## Releasing a new version
 
 Tskit maintains separate versioning for the C API and Python package, each has its own
@@ -1113,6 +1152,8 @@ e.g. 1.0.0b1. Update the Python CHANGELOG.rst, ensuring that all significant
 changes since the last release have been listed. Comparing
 `git log --follow --oneline -- python`
 with `git log --follow --oneline -- python/CHANGELOG.rst` may help here.
+Once the version number is updated run the benchmarks as detailed above and
+commit the `bench-results.json/html` files.
 Once this PR is merged, push a tag to github:
 
 ```bash