-
Notifications
You must be signed in to change notification settings - Fork 738
Closed
Labels
a/scalabilityArea: related to scalabilityArea: related to scalabilityc/storage/safekeeperComponent: storage: safekeeperComponent: storage: safekeeper
Description
Following #9335, move preprocessing and shard filtering from pageserver to safekeeper.
This also requires protocol changes, where the subscriber specifies a shard ID, and receives preprocessed and filtered per-shard WAL records.
Protocol changes:
- Transmit ShardIndex to safekeeper when subscribing for WAL
- Change WAL send protocol to send ephemeral file ready buffer and list of metadata modification instructions
- Maintain application level checksumming. Currently, each wal record includes a crc, which is checked after the record is reconstructed on the pagserver side.
Metadata
Metadata
Assignees
Labels
a/scalabilityArea: related to scalabilityArea: related to scalabilityc/storage/safekeeperComponent: storage: safekeeperComponent: storage: safekeeper