[NOT-64] Weeknotes individual update: October 30, 2023

[dtbuchholz]

Postgres Logical Replication in Rust

by Avichal Pandey

Basin uses Postgres' streaming replication protocol to read the changes on a database and archive them in Filecoin. The basin cli is written in Go. It leverages mature libraries, especially pgx and pglogrepl. The latter offers higher-level abstractions to communicate with a Postgres server to extract WAL updates.

While trying to write a streaming replication client in Rust, I found that the logical replication libraries are not as mature. The rust-postgres project has a couple of open pull requests. These will add convenient higher-level methods for working with logical replication, but these still need to be merged.

Materialize's fork of rust-postgres has these features merged. It is stable and used by MaterializedInc in production. However, it has minimal documentation. The logical decoding example by @seddonm1 is an excellent starting point for someone trying to build apps that use Postgres streaming replication capabilities. It uses decoderbuf to extract the WAL messages.

I want to share a repository that uses Materialize's fork of rust-postgres mentioned above. It uses the wal2json plugin to extract and read WAL messages from Postgres and replicate them on an embedded DuckDB instance. The code illustrates the following:

1. creating publications
2. creating and using replication slots
3. extracting JSON-encoded WAL messages
4. sending timely commit and keepalive messages to the Postgres server
5. using an embedded duck db instance to replicate changes

_{From SyncLinear.com | NOT-64}