Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new WAL #25144

Closed
Tracked by #25142
pauldix opened this issue Jul 11, 2024 · 1 comment
Closed
Tracked by #25142

Implement new WAL #25144

pauldix opened this issue Jul 11, 2024 · 1 comment
Labels

Comments

@pauldix
Copy link
Member

pauldix commented Jul 11, 2024

The server will now have a single logical WAL, this issue tracks implementing this new WAL, a precursor to getting the server refactored to use it. It will have the following properties:

  • The WAL will be written to a configured object store
  • The WAL will have an enum of Entry that will have different operations (writelp, write parquet, write snapshot file, etc.)
  • Each entry will have a monotonically increasing u64 sequence number
  • The WAL will be comprised of individual files, each being identified by their own sequence number
  • Writes to the WAL will be buffered up on a flush interval (default 1s), after which a file will be written with its sequence number as the filename
  • After N + N/2 files have been created, a snapshot will be run (N is configurable, default is 600 (i.e. snapshot after we have 900 files with the goal of reaping the first 600. Thus snapshotting every 10 minutes by default)
  • Snapshot should be an interface that the WAL gets initialized with
  • Once the snapshot is complete and successful, remove the first 600 files

The WAL should be able to be initialized empty or pointed to an existing object store. All WAL files should be read and replayed in order. There will be an interface to replay WAL entries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants