docs & scripts: external build (#78)

Close #74

---------

Signed-off-by: cutecutecat <junyuchen@tensorchord.ai>

README.md

@@ -141,6 +141,37 @@ You can check the indexing progress by querying the `pg_stat_progress_create_ind
 SELECT phase, round(100.0 * blocks_done / nullif(blocks_total, 0), 1) AS "%" FROM pg_stat_progress_create_index;
 ```
 
+### External Index Precomputation
+
+Unlike pure SQL, an external index precomputation will first do clustering outside and insert centroids to a PostgreSQL table. Although it might be more complicated, external build is definitely much faster on larger dataset (>5M).
+
+To get started, you need to do a clustering of vectors using `faiss`, `scikit-learn` or any other clustering library.
+
+The centroids should be preset in a table of any name with 3 columns:
+- id(integer): id of each centroid, should be unique
+- parent(integer, nullable): parent id of each centroid, should be NULL for normal clustering
+- vector(vector): representation of each centroid, `pgvector` vector type
+
+And example could be like this:
+
+```sql
+-- Create table of centroids
+CREATE TABLE public.centroids (id integer NOT NULL UNIQUE, parent integer, vector vector(768));
+-- Insert centroids into it
+INSERT INTO public.centroids (id, parent, vector) VALUES (1, NULL, '{0.1, 0.2, 0.3, ..., 0.768}');
+INSERT INTO public.centroids (id, parent, vector) VALUES (2, NULL, '{0.4, 0.5, 0.6, ..., 0.768}');
+INSERT INTO public.centroids (id, parent, vector) VALUES (3, NULL, '{0.7, 0.8, 0.9, ..., 0.768}');
+-- ...
+
+-- Create index using the centroid table
+CREATE INDEX ON gist_train USING vchordrq (embedding vector_l2_ops) WITH (options = $$
+[build.external]
+table = 'public.centroids'
+$$);
+```
+
+To simplify the workflow, we provide end-to-end scripts for external index pre-computation, see [scripts](./scripts/README.md#run-external-index-precomputation-toolkit).
+
 ### Installing From Source
 Install pgrx according to [pgrx's instruction](https://github.com/pgcentralfoundation/pgrx?tab=readme-ov-file#getting-started).
 ```bash

scripts/README.md

@@ -0,0 +1,87 @@
+## Build Docker
+
+```shell
+sudo apt install -y build-essential libreadline-dev zlib1g-dev flex bison libxml2-dev libxslt-dev libssl-dev libxml2-utils xsltproc ccache pkg-config clang
+cargo install --locked cargo-pgrx
+cargo pgrx init
+cargo build --package vchord --lib --features pg16 --target x86_64-unknown-linux-gnu --profile opt
+./tools/schema.sh --features pg16 --target x86_64-unknown-linux-gnu --profile opt
+
+export SEMVER="0.0.0"
+export VERSION="16"
+export ARCH="x86_64"
+export PLATFORM="amd64"
+export PROFILE="opt"
+./tools/package.sh
+
+docker build -t vchord:pg16-latest --build-arg PG_VERSION=16 -f ./docker/Dockerfile .
+```
+
+## Run Instance
+
+```shell
+docker run --name vchord -e POSTGRES_PASSWORD=123 -p 5432:5432 -d vchord:pg16-latest
+
+PGPASSWORD=123 psql -h 127.0.0.1 -U postgres -c "CREATE USER bench WITH PASSWORD '123';"
+PGPASSWORD=123 psql -h 127.0.0.1 -U postgres -c "ALTER ROLE bench SUPERUSER;"
+```
+
+## Run External Index Precomputation Toolkit
+
+1. Install requirements
+
+```shell
+# PYTHON = 3.11
+# When using CPU to train k-means clustering
+conda install conda-forge::pgvector-python numpy pytorch::faiss-cpu conda-forge::psycopg h5py tqdm
+# or
+pip install pgvector-python numpy faiss-cpu psycopg h5py tqdm
+
+# When using GPU to train k-means clustering
+conda install conda-forge::pgvector-python numpy pytorch::faiss-gpu conda-forge::psycopg h5py tqdm
+```
+
+1. Prepare dataset in `hdf5` format
+
+   - If you already have your vectors stored in `PostgreSQL` using `pgvector`, you can export them to a local file by:
+     ```shell
+     python script/dump.py -n [table name] -c [column name] -d [dim] -o export.hdf5
+     ```
+
+   - If you don't have any data, but would like to give it a try, you can choose one of these datasets:
+
+     ```shell
+     wget http://ann-benchmarks.com/sift-128-euclidean.hdf5 # num=1M dim=128 metric=l2
+     wget http://ann-benchmarks.com/gist-960-euclidean.hdf5 # num=1M dim=960 metric=l2
+     wget https://myscale-datasets.s3.ap-southeast-1.amazonaws.com/laion-5m-test-ip.hdf5 # num=5M dim=768 metric=dot
+     wget https://myscale-datasets.s3.ap-southeast-1.amazonaws.com/laion-20m-test-ip.hdf5 # num=20M dim=768 metric=dot
+     wget https://myscale-datasets.s3.ap-southeast-1.amazonaws.com/laion-100m-test-ip.hdf5 # num=100M dim=768 metric=dot
+     ```
+
+2. Preform clustering of centroids from vectors
+
+   ```shell
+   # For small dataset size from 1M to 5M
+   python script/train.py -i [dataset file(export.hdf5)] -o [centroid filename(centroid.npy)] -lists [lists] -m [metric(l2/cos/dot)]
+   # For large datasets size, 5M to 100M in size, use GPU and mmap chunks
+   python script/train.py -i [dataset file(export.hdf5)] -o [centroid filename(centroid.npy)] --lists [lists] -m [metric(l2/cos/dot)] -g --mmap
+   ```
+
+   `lists` is the number of centroids for clustering, and a typical value could range from: 
+   $$
+   4*\sqrt{len(vectors)} \le lists \le 16*\sqrt{len(vectors)}
+   $$
+
+3. To insert vectors and centroids into the database, and then create an index 
+
+   ```shell
+   python script/index.py -n [table name] -i [dataset file(export.hdf5)] -c [centroid filename(centroid.npy)] -m [metric(l2/cos/dot)] -d [dim]
+   ```
+
+4. Let's start our tour to check the benchmark result of VectorChord
+
+   ```shell
+   python script/bench.py -n [table name] -i [dataset file(export.hdf5)] -m [metric(l2/cos/dot)] -p [database password] --nprob 100 --epsilon 1.0
+   ```
+
+    Larger `nprobe` and `epsilon` will have a more precise query but at a slower speed.