Repository navigation
docs: correct WAL sync-mode claims and document all-null column handling - #19
Merged
Merged
Conversation
Follows Basekick-Labs/arc#565. WAL sync modes: the docs claimed fdatasync was "50% faster than fsync". That was never measurable, because both modes called the same full Sync() until arc#305 -- and the benchmark table reflected it, showing fdatasync as SLOWER than fsync (7.5M vs 7.7M rec/s), which is what measuring two identical code paths plus noise looks like. Corrected to say what is actually true: syncs are batched on a 100ms ticker rather than performed per write, so all three modes land within noise of each other on throughput. The sync mode selects durability semantics, not speed. Collapsed the per-mode benchmark rows into a single "WAL enabled" row, and removed the troubleshooting step that advised switching to fdatasync to fix slow throughput -- that advice cannot help. Also documented platform support: fdatasync(2) is Linux-only; macOS and Windows fall back to a full fsync and now log that once at startup. Added a note that macOS Sync() maps to fsync(2), which does not flush the drive's write cache. MessagePack API: documented that a column may be entirely null within a batch and is preserved rather than dropped (arc#337), that a later batch with real values determines the type, and that time is the exception -- a null or non-numeric time is rejected with 400, verified against a running server.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Documentation follow-up to Basekick-Labs/arc#565.
WAL sync modes were documented inaccurately
The docs claimed fdatasync was "50% faster than
fsync". That was never measurable — until arc#305, both modes called the same fullSync().The benchmark table showed it plainly and nobody noticed:
fdatasync listed as slower than fsync is exactly what measuring two identical code paths plus noise looks like.
Corrected:
fdatasync(2)is Linux-only; macOS and Windows fall back to a fullfsyncand now log it once at startup (fdatasync is unavailable on this platform; using full fsync instead), withfdatasync_supportedin the startup log.Sync()maps tofsync(2), which does not flush the drive's own write cache —F_FULLFSYNCwould be required. Relevant for local development, not Linux production.MessagePack null handling
Documented the arc#337 behavior on
POST /api/v1/write/msgpack:timeis the exception: a null or non-numerictimeis rejected with400 Bad Request, because a non-timestamptimecolumn makes the partition un-compactable.I verified the
400against a running server rather than assuming it — the rejection happens synchronously innormalizeTimestamps, so it is a real HTTP error and not an async flush failure:Note
The corrected throughput figures are the existing published numbers with the misleading per-mode split removed — I did not re-benchmark. If you want fresh numbers for 26.09.1, that's worth a separate pass on a quiet machine.