Skip to content

safekeeper: lift WAL preprocessing and filtering from pageserver #9336

@erikgrinaker

Description

@erikgrinaker

Following #9335, move preprocessing and shard filtering from pageserver to safekeeper.

This also requires protocol changes, where the subscriber specifies a shard ID, and receives preprocessed and filtered per-shard WAL records.

Protocol changes:

  • Transmit ShardIndex to safekeeper when subscribing for WAL
  • Change WAL send protocol to send ephemeral file ready buffer and list of metadata modification instructions
  • Maintain application level checksumming. Currently, each wal record includes a crc, which is checked after the record is reconstructed on the pagserver side.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions