turbo-persistence: SST write path I/O and allocation optimizations#90542
Open
lukesandberg wants to merge 1 commit intocanaryfrom
Open
turbo-persistence: SST write path I/O and allocation optimizations#90542lukesandberg wants to merge 1 commit intocanaryfrom
lukesandberg wants to merge 1 commit intocanaryfrom
Conversation
nextjs-bot
added
created-by: Turbopack team
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
lukesandberg
changed the title
Merging this PR will not alter performance
Comparing 
Footnotes
|
Collaborator
Tests Passed |
lukesandberg
commented
Feb 25, 2026
Comment on lines
+545
to
+553
| self.parallel_scheduler | ||
| .try_parallel_for_each(&new_sst_files, |(_, file)| { | ||
| file.sync_data()?; | ||
| anyhow::Ok(()) | ||
| })?; | ||
| self.parallel_scheduler | ||
| .try_parallel_for_each(&new_blob_files, |(_, file)| { | ||
| file.sync_data()?; | ||
| anyhow::Ok(()) |
Contributor
Author
There was a problem hiding this comment.
can we just concat the iterators and do one parallel for each
Collaborator
Stats from current PR✅ No significant changes detected📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles: **399 kB** → **399 kB** ✅ -21 B80 files with content-based hashes (individual files not comparable between builds) Server Middleware
Build DetailsBuild Manifests
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URL |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Targeted optimizations to the turbo-persistence SST write path, informed by profiling the compaction benchmark with samply.
Changes
sync_all→sync_data+ parallel syncs (db.rs)sync_data()instead ofsync_all()for all file syncs incommit()— avoids syncing metadata (timestamps) we don't need for crash safetytry_parallel_for_eachsince they're independent I/O operationsLarger BufWriter capacity (
static_sorted_file_builder.rs,meta_file_builder.rs)Software position tracking (
static_sorted_file_builder.rs)file.stream_position()with a computed value from block offsetsBufWriterdoes not overrideSeek::stream_position, so calling it flushes the entire write buffer then issues anlseeksyscall — defeating the larger buffer from change Exposing css as next/css #2Dict-less LZ4 compression uses
compress_to_vec(compression.rs)lz4::compress_to_vecwhich uses a thread-localExtStatewith fast reset — no allocationCompressor::with_dictonly for dictionary-compressed key blocks where it's actually neededBetter
KeyBlockBuilderreservation (static_sorted_file_builder.rs)current_block_sizeas a size hint toKeyBlockBuilder::newso the initialVec::reserveis accurate, reducing reallocation during block buildingBenchmark Results
Compared
canaryvs this branch usingcargo bench -p turbo-persistence:Note: The benchmarks run single-threaded, so the parallel
sync_dataoptimization (addressing 19% of profiled time) has no visibility here. Real-world impact should be larger, particularly for commits with multiple SST files where fsyncs can be pipelined by the OS.