Skip to content

Commit d81e8d4

Browse files
levkklevkk
authored
feat: document new reshard command and schema sync (#55)
1 parent 7436248 commit d81e8d4

File tree

3 files changed

+174
-63
lines changed

3 files changed

+174
-63
lines changed

‎docs/features/sharding/resharding/databases.md‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,19 +4,17 @@ icon: material/database-plus-outline
44

55
# New databases
66

7-
PgDog's strategy for resharding Postgres databases is to create a new, independent cluster of machines and move data over to it in real-time. Creating new databases is environment-specific, and PgDog doesn't currently automate this step.
7+
PgDog's strategy for resharding Postgres databases is to create a new, independent cluster of machines and move data over to it in real-time. Creating new databases is environment-specific, and PgDog doesn't currently automate this step[^1].
88

99
## Requirements
1010

1111
New databases should be **empty**: don't migrate your [table definitions](schema.md) or [data](hash.md). These will be taken care of automatically by PgDog.
1212

1313
### Database users
1414

15-
Since PgDog was built to work in cloud-managed environments, like AWS RDS, we don't usually have access to the `pg_shadow` view, which contains password hashes. Therefore, tools like [`pg_dumpall`](https://www.postgresql.org/docs/current/app-pg-dumpall.html) aren't able to operate correctly, and we can't automatically migrate users to the new database.
15+
Since PgDog was built to work in cloud-managed environments, like AWS RDS, it doesn't usually have access to the `pg_shadow` view, which contains password hashes. Therefore, tools like [`pg_dumpall`](https://www.postgresql.org/docs/current/app-pg-dumpall.html) aren't able to operate correctly, and we can't automatically migrate users to the new database.
1616

17-
For this reason, migrating users to the new database cluster is currently **not supported** and is the responsibility of the operator.
18-
19-
Make sure to create all the necessary Postgres users and roles before proceeding to the [next step](schema.md).
17+
For this reason, migrating users to the new database cluster is currently **not supported** and is the responsibility of the operator. Make sure to create all the necessary Postgres users and roles before proceeding to [schema synchronization](schema.md).
2018

2119
## Multiple Postgres databases
2220

@@ -25,5 +23,8 @@ If you are operating multiple Postgres databases on the same database server, th
2523
## Next steps
2624

2725
{{ next_steps_links([
28-
("Schema sync", "schema.md", "Synchronize schema across new shards before moving data."),
26+
("Schema sync", "schema.md", "Synchronize schema entities like table and index definitions to the new shards before moving data. This will make sure logical replication works as expected."),
2927
]) }}
28+
29+
30+
[^1]: We are building a control plane for the [Enterprise Edition](../../../enterprise_edition/index.md) which will take care of creating databases, with support for various cloud vendors (e.g., AWS RDS, Azure SQL, etc.).

‎docs/features/sharding/resharding/index.md‎

Lines changed: 32 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -4,32 +4,51 @@ icon: material/set-split
44

55
# Resharding Postgres
66

7-
!!! note
8-
This feature is a work in progress. Support for resharding with logical replication was started in [#279](https://github.com/pgdogdev/pgdog/pull/279).
7+
!!! note "Work in progress"
8+
This feature is in active development. Support for resharding with logical replication was started in [#279](https://github.com/pgdogdev/pgdog/pull/279) and
9+
received major improvements in [#784](https://github.com/pgdogdev/pgdog/pull/784).
910

10-
Resharding changes the number of shards in an existing database cluster, in order to add or remove capacity. To make this less impactful on production operations, PgDog's strategy for resharding is to create a new database cluster and reshard data in-flight, while moving it to the new databases.
11+
Resharding changes the number of shards in an existing database cluster in order to add or remove capacity. To make this less impactful on production operations, PgDog's strategy for resharding is to create a new database cluster and reshard data while moving it to the new databases.
1112

1213
To make this an online process, with zero downtime or data loss, PgDog hooks into the logical replication protocol used by PostgreSQL and reroutes messages between nodes to create and update rows in real-time.
1314

14-
<center>
15-
<img src="/images/resharding-arch-1.png" width="90%" height="auto" alt="Mirroring">
15+
<center style="margin-top: 40px;">
16+
<img src="/images/resharding-arch-1.png" width="80%" height="auto" alt="Mirroring">
1617
</center>
1718

18-
## Step by step
19+
## Resharding process
1920

20-
The resharding process is composed of four independent operations:
21+
The resharding process is composed of four independent operations. The first one is currently the responsibility of the user, while the remaining 3 are automated by PgDog:
2122

22-
1. #### [Create new databases](databases.md)
23-
2. #### [Synchronize schema](schema.md)
24-
3. #### [Move data](hash.md)
25-
4. #### [Cutover traffic](cutover.md)
23+
| Operation | Description |
24+
|-|-|
25+
| [Create new cluster](databases.md) | Create a new set of empty databases that will be used for storing data in the new, sharded cluster. |
26+
| [Schema synchronization](schema.md) | Replicate table and index definitions to the new shards, making sure the new cluster has the same schema as the old one. |
27+
| [Move & reshard data](hash.md) | Copy data using logical replication, while redistributing rows in-flight between new shards. |
28+
| [Cutover traffic](cutover.md) | Make the new cluster service both reads and writes from the application, without taking downtime. |
29+
30+
While each step can be executed separately by the operator, PgDog provides an [admin database](../../../administration/index.md) command to perform online resharding and traffic cutover steps in a completely automated fashion:
31+
32+
```
33+
RESHARD <source> <destination> <publication>;
34+
```
2635

27-
Steps two and three are automated by PgDog, while their orchestration is currently the responsibility of the user.
36+
The `<source>` and `<destination>` parameters accept the name of the source and destination databases respectively. The `<publication>` parameter expects the name of the Postgres [publication](schema.md#publication) for the tables that need to be resharded.
37+
38+
!!! note "Traffic cutover"
39+
Traffic cutover requires careful synchronization to avoid data loss and a split-brain situation. The `RESHARD` command supports this for **single node** PgDog deployments only. The [Enterprise Edition](../../../enterprise_edition/index.md) provides a control plane, which supports traffic cutover with multiple PgDog containers.
2840

2941
## Terminology
3042

3143
| Term | Description |
3244
|-|-|
3345
| Source database | The database cluster that's being resharded and contains all data and table definitions. |
34-
| Destination database | The database cluster with the new sharding configuration, where the data will be copied from the source database. |
46+
| Destination database | The database cluster with the new sharding configuration, to where the data will be copied from the source database. |
3547
| Logical replication | Replication protocol available to PostgreSQL databases since version 10. |
48+
49+
## Next steps
50+
51+
{{ next_steps_links([
52+
("Schema sync", "schema.md", "Synchronize table, index and other schema entities between the source and destination databases."),
53+
("Move data", "hash.md", "Redistribute data between shards using the configured sharding function. This happens without downtime and keeps the shards up-to-date with the source database until traffic cutover."),
54+
]) }}

‎docs/features/sharding/resharding/schema.md‎

Lines changed: 135 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -3,17 +3,52 @@ icon: material/database-edit-outline
33
---
44
# Schema sync
55

6-
PgDog can copy tables, indexes and other entities from your production database to the new, sharded database automatically. This is faster than using `pg_dump`, because we separate this process into two parts:
6+
PostgreSQL logical replication requires that tables on both the source and destination databases contain the same columns, with compatible data types. PgDog takes care of this, by using `pg_dump` under the hood, and re-creating table and index definitions, in an optimal order, on the new shards.
77

8-
1. [Create tables](#tables-and-primary-keys), primary key indexes, and sequences
9-
2. Create [secondary indexes](#secondary-indexes)
10-
3. Move [sequence](#sequences) values
8+
### Synchronization phases
119

12-
The create tables step needs to be performed first, before [copying data](hash.md). The second step is performed once the data sync is almost complete.
10+
The schema synchronization process is composed of 4 distinct steps, all of which are executed automatically by PgDog during resharding:
1311

14-
## CLI
12+
| Phase | Description |
13+
|-|-|
14+
| [Pre-data](#pre-data-phase) | Create identical tables on all shards along with the primary key constraint (and index). Secondary indexes are _not_ created yet. |
15+
| [Post-data](#post-data-phase) | Create secondary indexes on all tables and shards. This is done after [moving data](hash.md), as a separate step, because it's considerably faster to create indexes on whole tables than while inserting individual rows. |
16+
| [Cutover](#cutover) | This step is executed during traffic cutover, while application queries are blocked from executing on the database. |
17+
| Post-cutover | This step makes sure the rollback database cluster can handle reverse logical replication. |
18+
19+
## Performing the sync
20+
21+
Schema synchronization can be performed using one of two methods:
22+
23+
1. Using an [admin database](../../../administration/index.md) command
24+
2. Using a CLI command
25+
26+
### Admin database command
27+
28+
The admin database provides an easy way to execute commands, without having to spawn a new PgDog process. The schema synchronization command has the following syntax:
29+
30+
```
31+
SCHEMA_SYNC <phase> <source database> <destination database> <publication>;
32+
```
33+
34+
The `<phase>` argument accepts the following values:
35+
36+
| Phase | Description |
37+
|-|-|
38+
| `PRE` | Perform the pre-data schema synchronization phase. |
39+
| `POST` | Perform the post-data schema synchronization phase. |
1540

16-
PgDog has a command line interface you can call by running it directly. Schema sync is controlled by a CLI command:
41+
##### Example
42+
43+
To perform schema synchronization for the pre-data step from database `"prod"` to database `"prod_sharded"` and the `"all_tables"` publication, execute the following command:
44+
45+
```
46+
SCHEMA_SYNC PRE prod prod_sharded all_tables;
47+
```
48+
49+
### CLI
50+
51+
PgDog has a command line interface you can call by running the `pgdog` executable directly. Schema sync has its own CLI command with the following arguments:
1752

1853
```
1954
pgdog schema-sync \
@@ -22,47 +57,105 @@ pgdog schema-sync \
2257
--publication <publication>
2358
```
2459

25-
Required (*) and optional parameters for this command are as follows:
26-
2760
| Parameter | Description |
2861
|-|-|
29-
| `--from-database`* | The name of the source database in `pgdog.toml`. |
30-
| `--to-database`* | The name of the destination database in `pgdog.toml`. |
31-
| `--publication`* | The name of the Postgres table [publication](#publication) with the tables you want to sync. |
32-
| `--dry-run` | Print the SQL statements that will be executed on the destination database and exit. |
33-
| `--ignore-errors` | Execute SQL statements and ignore any errors. |
34-
| `--data-sync-complete` | Run the second step to create secondary indexes and sequences. |
35-
| `--cutover` | Run the cutover step to move sequence values. |
62+
| `--from-database` * | The name of the source database in [`pgdog.toml`](../../../configuration/pgdog.toml/databases.md). |
63+
| `--to-database` * | The name of the destination database in [`pgdog.toml`](../../../configuration/pgdog.toml/databases.md). |
64+
| `--publication` * | The name of the PostgreSQL [publication](#publication) with the tables you want to synchronize. |
65+
| `--dry-run` | Only print the SQL statements that will be executed on the destination database and exit. |
66+
| `--ignore-errors` | Ignore any errors caused by executing any of the schema synchronization SQL statements. |
67+
| `--data-sync-complete` | Run the post-data step to create secondary indexes and sequences. |
68+
| `--cutover` | Run the cutover step during traffic cutover. |
3669

37-
## Tables and primary keys
70+
## Pre-data phase
3871

39-
The first step in the schema sync copies over tables and their primary key indexes from the source database to the new, resharded cluster. This has to be done separately, because Postgres's logical replication only copies data and doesn't manage table schemas.
72+
The pre-data phase takes care of replicating the following Postgres schema entities:
4073

41-
### Primary keys
74+
1. Table schemas (e.g. `CREATE SCHEMA`)
75+
2. Table definitions, with identical columns and data types (e.g., `CREATE TABLE`)
76+
3. Custom types and domains (e.g., `CREATE TYPE`, `CREATE DOMAIN`)
77+
4. Extensions (e.g., `CREATE EXTENSION pgvector`)
78+
5. Primary key constraints and corresponding unique indexes (e.g., `PRIMARY KEY (id)`)
79+
6. Table publications (e.g., `CREATE PUBLICATION`)
4280

43-
A primary key constraint is **required** on all tables for logical replication to work correctly. Without a unique index identifying each row in a table, logical replication is not able to perform `UPDATE` and `DELETE` commands.
81+
!!! note "Primary key requirement"
82+
PgDog requires that _all_ tables that are being resharded contain a **primary key** constraint. This is important for logical replication
83+
and guarantees that `UPDATE` and `DELETE` statements are replicated correctly between the source database and the new shards.
4484

45-
Before starting the resharding process for your database, double-check that you have primary keys on all your tables.
85+
Since the pre-data phase creates only empty tables, it can be executed very quickly even for databases with a larger number of tables, extensions and custom data types.
4686

4787
### Publication
4888

49-
Since PgDog is using logical replication to move and reshard data, a [publication](https://www.postgresql.org/docs/current/sql-createpublication.html) for the relevant tables needs to be created on the source database.
89+
Since PgDog is using logical replication to move and reshard data, a [publication](https://www.postgresql.org/docs/current/sql-createpublication.html) for the relevant tables needs to be created on the source database beforehand. The simplest way to do this is to run the following command:
90+
91+
```postgresql
92+
CREATE PUBLICATION pgdog FOR ALL TABLES;
93+
```
94+
95+
This will make sure all tables and schemas in your database are copied and resharded into the destination database cluster.
96+
97+
##### Example
98+
99+
=== "Admin database"
100+
```
101+
SCHEMA_SYNC PRE prod prod_sharded all_tables;
102+
```
103+
=== "CLI"
104+
```
105+
pgdog schema-sync \
106+
--from-database prod \
107+
--to-database prod_sharded \
108+
--publication all_tables
109+
```
110+
111+
## Post-data phase
112+
113+
The post-data phase is performed after the [data copy](hash.md) is complete and tables have been synchronized with logical replication. Its job is to create all secondary indexes (e.g., `CREATE INDEX`).
50114

51-
The simplest way to do this is to run the following command:
115+
This step is performed after copying data because it makes the copy process considerably faster: Postgres doesn't need to update several indexes while writing rows into the tables.
52116

53-
=== "Source database"
54-
```postgresql
55-
CREATE PUBLICATION pgdog FOR ALL TABLES;
117+
##### Example
118+
119+
=== "Admin database"
120+
```
121+
SCHEMA_SYNC POST prod prod_sharded all_tables;
122+
```
123+
=== "CLI"
56124
```
125+
pgdog schema-sync \
126+
--from-database prod \
127+
--to-database prod_sharded \
128+
--publication all_tables
129+
--data-sync-complete
130+
```
131+
132+
### Tracking progress
57133

58-
This will make sure _all_ tables in your database will be copied and resharded into the destination database cluster.
134+
Since creating indexes on large tables can take some time, PgDog provides an admin database command to monitor the progress:
59135

60-
!!! note "Multiple schemas"
61-
If you're using schemas other than `public`, create them on the destination database before running the schema sync.
136+
=== "Admin command"
137+
```
138+
SHOW SCHEMA_SYNC;
139+
```
140+
=== "Output"
141+
```
142+
-[ RECORD 1 ]+-----------------------------------------------------------------------------------------------
143+
database | pgdog
144+
user | pgdog
145+
shard | 1
146+
kind | index
147+
sync_state | running
148+
started_at | 2026-01-15 10:32:01.042 UTC
149+
elapsed | 3s
150+
elapsed_ms | 3012
151+
table_schema | public
152+
table_name | users
153+
sql | CREATE INDEX CONCURRENTLY IF NOT EXISTS "users_email_idx" ON "public"."users" USING btree ("email")
154+
```
62155

63156
### Schema admin
64157

65-
Schema sync creates tables, indexes, and other entities on the destination database. To make sure that's done with a user with sufficient privileges (e.g., `CREATE` permission on the database), you need to add it to [`users.toml`](../../../configuration/users.toml/users.md) and mark it as the schema administrator:
158+
Schema sync creates tables, indexes, and other entities on the destination database. To make sure that this is done with a user with sufficient privileges (e.g., `CREATE` and `REPLICATION` permissions on the database), make sure to add such a user to [`users.toml`](../../../configuration/users.toml/users.md) and mark it as the schema administrator:
66159

67160
```toml
68161
[[users]]
@@ -72,11 +165,19 @@ password = "hunter2"
72165
schema_admin = true
73166
```
74167

75-
PgDog will use that user to connect to the source and destination databases, so make sure to specify one for both of them.
168+
PgDog will use this user to connect to the source and destination databases, so make sure to specify one for both databases in the configuration.
169+
170+
## Cutover
171+
172+
During the cutover, PgDog will execute last minute schema synchronization commands to make sure the destination sharded cluster works as expected. This involves putting back constraints that were removed for logical replication to work and moving sequence values.
173+
174+
## Post-cutover
76175

77-
### `pg_dump` version
176+
The post-cutover phase makes sure the source database can accept logical replication streams from the new shards. This maintains the old database in sync, in case the operator decides to roll back the resharding process.
78177

79-
PgDog is using `pg_dump` under the hood to export schema definitions. Postgres requires the version of `pg_dump` and the Postgres server to be identical. Our [Docker image](../../../installation.md) comes with `pg_dump` for PostgreSQL 16, but your database server may run a different version.
178+
## Dependencies
179+
180+
PgDog is using `pg_dump` under the hood to export table, schema and index definitions. PostgreSQL servers typically require that the version of `pg_dump` and the version of the server are identical. Our [Docker image](../../../installation.md) comes with `pg_dump` for PostgreSQL 16 by default, but your database server may run a different version.
80181

81182
Before proceeding, make sure to install the correct version of `pg_dump` for your source database. If you have multiple versions of `pg_dump` installed on the same host, you can specify the path to the right one in `pgdog.toml`:
82183

@@ -85,18 +186,8 @@ Before proceeding, make sure to install the correct version of `pg_dump` for you
85186
pg_dump_path = "/path/to/pg_dump"
86187
```
87188

88-
## Secondary indexes
89-
90-
This step is performed after [data sync](hash.md) is complete. Running this step will create secondary indexes on all your tables, which will take some time, depending on the number of indexes in your schema.
91-
92-
## Sequences
93-
94-
This steps is performed during the cutover stage once both schema sync and data sync are complete. The source database is no longer accepting writes and we are ready to move them to the destination.
95-
96-
This step will calculate the `MAX(column) + 1` values for all table sequences and set them on the respective columns.
97-
98189
## Next steps
99190

100191
{{ next_steps_links([
101-
("Move data", "hash.md", "Redistribute data across shards using hash-based resharding."),
192+
("Move data", "hash.md", "Redistribute data between shards using the configured sharding function. This happens without downtime and keeps the shards up-to-date with the source database until traffic cutover."),
102193
]) }}

0 commit comments

Comments
 (0)