commitlog,durability: Support preallocation of disk space#3437
Merged
Conversation
When a new commitlog segment is created, allocate disk space for it up to the maximum segment size. Also do this when resuming writes to an existing segment, such that segments created without preallocation will allocate as well when the database is opened. Preallocation is gated behind the feature "fallocate", because it is not always desirable to preallocate, e.g. for local `standalone` users. The feature can only be enabled on Linux targets, because allocation is done using the Linux-specific `fallocate(2)` system call. Unlike `ftruncate(2)` or the portable `posix_fallocate(3)`, `fallocate(2)` supports allocating disk space without zeroing. This is currently required, because the commitlog format does not handle padding bytes. The commitlog refuses writes if not enough space can be allocated. For commitlogs that were created without preallocation, this means that the commitlog cannot even be opened. The local durability impl will crash if it detects that the commitlog is unable to allocate enough space. This means that a database will eventually crash and unable to start in an out-of-space situation. Allocated space is not included in the reported size of the commitlog. Instead, allocated blocks are reported separately.
Shubham8287
approved these changes
Nov 7, 2025
Contributor
Shubham8287
left a comment
Shubham8287
left a comment
There was a problem hiding this comment.
Sorry, I reviewed this PR long back but completely forgot to hit the button.
Tests seems good.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a new commitlog segment is created, allocate disk space for it up
to the maximum segment size. Also do this when resuming writes to an
existing segment, such that segments created without preallocation will
allocate as well when the database is opened.
Preallocation is gated behind the feature "fallocate", because it is not
always desirable to preallocate, e.g. for local
standaloneusers.The feature can only be enabled on Linux targets, because allocation is
done using the Linux-specific
fallocate(2)system call.Unlike
ftruncate(2)or the portableposix_fallocate(3),fallocate(2)supports allocating disk space without zeroing. This is currently
required, because the commitlog format does not handle padding bytes.
If not enough space can be allocated, the commitlog refuses writes. For
commitlogs that were created without preallocation, this means that the
commitlog cannot even be opened in this situation.
The local durability impl will crash if it detects that the commitlog is
unable to allocate enough space.
This means that a database will eventually crash and be unable to start in
an out-of-space situation.
Allocated space is not included in the reported size of the commitlog.
Instead, allocated blocks are reported separately.
Expected complexity level and risk
3 - Disk size monitoring may need to be adjusted.
Testing
spacetimedb_durability::Local]when there is insufficient space. The test performs I/O against a loop device.
repo::Memoryimpl so that it can run out of space. No test currentlyutilizes this, but existing tests assuming infinite space still pass.