docs: Explain how testing and CI works in more detail

README.md

@@ -41,6 +41,8 @@ Store encrypted data alongside your existing data:
   - [How is data encrypted in the database?](#how-is-data-encrypted-in-the-database)
 - [Helper packages](#helper-packages-and-examples)
 - [Releasing](#releasing)
+- [Developing](#developing)
+- [Testing](#testing)
 
 ---
 
@@ -393,20 +395,14 @@ To cut a [release](https://github.com/cipherstash/encrypt-query-language/release
 
 This will trigger the [Release EQL](https://github.com/cipherstash/encrypt-query-language/actions/workflows/release-eql.yml) workflow, which will build and attach artifacts to [the release](https://github.com/cipherstash/encrypt-query-language/releases/).
 
-## Testing
-
-There are tests for EQL for PostgreSQL versions 14–17.
-
-They easiest way to run them is in [GitHub Actions](https://github.com/cipherstash/encrypt-query-language/actions/workflows/test-eql.yml).
-
-### Running tests locally
+## Developing
 
 > [!IMPORTANT]
-> **Before you run the tests** you need to have this software installed:
+> **Before you follow the quickstart* you need to have this software installed:
 >  - [mise](https://mise.jdx.dev/) — see the [installing mise](#installing-mise) instructions
 >  - [Docker](https://www.docker.com/) — see Docker's [documentation for installing](https://docs.docker.com/get-started/get-docker/)
 
-To run tests locally:
+Local development quickstart:
 
 ``` shell
 # Clone the repo
@@ -416,17 +412,190 @@ cd encrypt-query-language
 # Install dependencies
 mise trust --yes
 
-# Start a postgres instance
-mise run postgres:up postgres-17 --extra-args "--detach --wait"
+# Build EQL installer and uninstaller, outputting to release/
+mise run build
+
+# Start a postgres instance (defaults to PostgreSQL 17)
+mise run postgres:up --extra-args "--detach --wait"
+
+# Run the tests (defaults to PostgreSQL 17)
+mise run test
+
+# Stop and remove all containers and networks
+mise run postgres:down
+```
+
+### Installing mise
+
+> [!IMPORTANT]
+> You must complete this step to set up a local development environment.
+
+Local development and task running in CI is managed through [mise](https://mise.jdx.dev/).
+
+To install mise:
+
+- If you're on macOS, run `brew install mise`
+- If you're on another platform, check out the mise [installation methods documentation](https://mise.jdx.dev/installing-mise.html#installation-methods)
+
+Then add mise to your shell:
+
+```shell
+# If you're running Bash
+echo 'eval "$(mise activate bash)"' >> ~/.bashrc
+
+# If you're running Zsh
+echo 'eval "$(mise activate zsh)"' >> ~/.zshrc
+```
+
+We use [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall) for faster installation of tools installed via `mise` and Cargo.
+We install `cargo-binstall` via `mise` when installing development and testing dependencies.
+
+> [!TIP]
+> We provide abbreviations for most of the commands that follow.
+> For example, `mise run postgres:setup` can be abbreviated to `mise r s`.
+> Run `mise tasks --extended` to see the task shortcuts.
+
+### How this project is organised
+
+Development is managed through [mise](https://mise.jdx.dev/), both locally and [in CI](https://github.com/cipherstash/encrypt-query-language/actions).
+
+mise has tasks for:
+
+- Building EQL install and uninstall scripts (`build`)
+- Starting and stopping PostgreSQL containers (`postgres:up`, `postgres:down`)
+- Running unit and integration tests (`test`, `reset`)
 
-# Run the tests
-mise run test --postgres 17
+These are the important files in the repo:
+
+```
+.
+├── mise.toml              <-- the main config file for mise
+├── tasks/                 <-- mise tasks
+├── sql/                   <-- The individual SQL components that make up EQL
+├── docs/                  <-- Tutorial, reference, and concept documentation
+├── tests/                 <-- Unit and integration tests
+│   ├── docker-compose.yml <-- Docker configuration for running PostgreSQL instances
+│   └── *.sql              <-- Individual unit and integration tests
+├── release/               <-- Build artifacts produced by the `build` task
+├── examples/              <-- Example uses of EQL in different languages
+└── playground/            <-- Playground enviroment for experimenting with EQL and CipherStash Proxy
+```
+
+## Testing
+
+There are tests for checking EQL against PostgreSQL versions 14–17, that verify:
+
+- Adding, removing, and modifying encrypted data and indexes
+- Validating, applying, and removing configuration for encrypted data and encrypted indexes
+- Validating schemas for EQL configuration, encrypted data, and encrypted indexes
+- Using PostgreSQL operators on encrypted data and indexes (`=`, `<>`, `@>`)
+
+The easiest way to run the tests [is in GitHub Actions](./.github/workflows/test-eql.yml):
+
+- Automatically whenever there are changes in the `sql/`, `tests/`, or `tasks/` directories
+- By manually running [the workflow](https://github.com/cipherstash/encrypt-query-language/actions/workflows/test-eql.yml)
+
+This is how the `test-eql.yml` workflow functions:
+
+```mermaid
+---
+title: Testing EQL
+---
+stateDiagram-v2
+    direction LR
+    classDef code font-family:monospace;
+
+
+    state "🧍 Human makes changes to EQL sources" as changes
+    state sources_fork <<fork>>
+    state sources_join <<join>>
+    state "sql/*.sql" as source_sql
+    state "tasks/**/*" as source_tasks
+    state "tests/**/*" as source_tests
+    state sources_changed <<choice>>
+
+    state "🛠️ Trigger GitHub Actions workflow test-eql.yml" as build_triggered
+    state "Matrix: Test EQL SQL components" as matrix
+    state "Test with Postgres 14" as pg14
+    state "Test with Postgres 15" as pg15
+    state "Test with Postgres 16" as pg16
+    state "Test with Postgres 17" as pg17
+    state "Check build results" as check
+    state if_state <<choice>>
+
+    changes --> sources_fork
+    sources_fork --> source_sql:::code
+    sources_fork --> source_tests:::code
+    sources_fork --> source_tasks:::code
+    source_sql --> sources_join
+    source_tests --> sources_join
+    source_tasks --> sources_join
+    sources_join --> source_changed_check
+    source_changed_check --> sources_changed
+    sources_changed --> build_triggered : Some changes
+    sources_changed --> [*]: No changes
+
+    state "Check source changes" as source_changed_check
+
+    [*] --> changes
+
+    build_triggered --> matrix
+
+    state fork_state <<fork>>
+        matrix --> fork_state
+        fork_state --> pg14
+        fork_state --> pg15
+        fork_state --> pg16
+        fork_state --> pg17
+
+    state join_state <<join>>
+        pg14 --> join_state
+        pg15 --> join_state
+        pg16 --> join_state
+        pg17 --> join_state
+
+    state "✅ Pass build" as build_pass
+    state "❌ Fail build" as build_fail
+    join_state --> check
+    check --> if_state
+    if_state --> build_pass: All success
+    if_state --> build_fail : Any failures
+    build_pass --> [*]
+    build_fail --> [*]
+```
+
+You can also [run the tests locally](#running-tests-locally) when doing local development.
+
+### Running tests locally
+
+> [!IMPORTANT]
+> **Before you run the tests locally** you need to [set up a local dev environment](#developing).
+
+To run tests locally with PostgreSQL 17:
+
+``` shell
+# Start a postgres instance (defaults to PostgreSQL 17)
+mise run postgres:up --extra-args "--detach --wait"
+
+# Run the tests (defaults to PostgreSQL 17)
+mise run test
 
 # Stop and remove all containers and networks
 mise run postgres:down
 ```
 
-You can run the same tasks for Postgres 14, 15, 16, and 17.
+You can run the same tasks for Postgres 14, 15, 16, and 17 by specifying arguments:
+
+```shell
+# Start a postgres 14 instance
+mise run postgres:up postgres-14 --extra-args "--detach --wait"
+
+# Run the tests against postgres 14
+mise run test --postgres 14
+
+# Stop postgres and remove all containers and networks
+mise run postgres:down
+```
 
 The configuration for the Postgres containers in `tests/docker-compose.yml`.
 
@@ -437,4 +606,5 @@ Limitations:
   `tests/docker-compose.yml`
 - **You can't run multiple Postgres containers at the same time.**
   All the containers bind to the same port (`7543`). If you want to run
-  multiple containers at the same time, you'll have to change the ports.
+  multiple containers at the same time, you have to change the ports by
+  editing `tests/docker-compose.yml`

tasks/build.sh

@@ -41,3 +41,12 @@ cat sql/666-rename_configuration_table.sql >> release/cipherstash-encrypt-uninst
 # remove the drop file
 rm release/cipherstash-encrypt-tmp-drop-install.sql
 rm release/cipherstash-encrypt-tmp-drop-uninstall.sql
+
+set +x
+echo
+echo '###############################################'
+echo "# ✅Build succeeded"
+echo '###############################################'
+echo
+echo 'Installer:   release/cipherstash-encrypt.sql'
+echo 'Uninstaller: release/cipherstash-encrypt-uninstall.sql'