Skip to content

fix(server): sqlite commits no longer fsync on the event loop - #60

Merged
alexcstark merged 1 commit into
mainfrom
fix/sqlite-synchronous-normal
Oct 11, 2026
Merged

alexcstark merged 1 commit into
mainfrom
fix/sqlite-synchronous-normal

Conversation

@alexcstark

Copy link
Copy Markdown
Owner

On lax6e the server's event loop stalls for 2–15s several times an hour, and the long ones drop every connected client. Today's 15.6s stall at 17:17 UTC disconnected the desktop.

The stalls were blocked waiting, not computing. The 30s sample around the 15.6s stall shows the loop busy (utilization 0.86) on only about 5s of CPU. I sampled the main thread every 200ms from /proc and caught the next stall (3178 ms logged at 17:49:58). The thread spent about 3.2s in state D inside syscall 74 (fsync), waiting on jbd2_log_wait_commit. That is SQLite's per-commit fsync, waiting on the ext4 journal of a busy, nearly full disk. CPU priority was not the issue: the service already has CPUWeight=1000, and its cgroup's CPU pressure is about 0.1%.

node:sqlite is synchronous and runs on the event loop. We never set synchronous, so SQLite used its default FULL, which fsyncs the WAL on every commit.

Fix

PRAGMA synchronous = NORMAL in layerSetup. This is SQLite's recommended setting for WAL:

  • Commits no longer fsync; only checkpoints do.
  • The database stays corruption-safe, and an app crash loses nothing.
  • Only an OS crash or power loss can roll back the last few commits.

There is a single read-write connection, and the other DatabaseSync openers are read-only, so this one pragma covers every writer.

Verification

  • vp test run src/persistence: 60/60 pass.
  • On lax6e's disk, under strace -c, 200 WAL commits made 209 fsyncs with FULL and 8 with NORMAL.

🤖 Generated with Claude Code (Opus 5.5)

@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:XS labels Oct 11, 2026
@github-actions

Copy link
Copy Markdown

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

ℹ️ The exact PR base did not have a successful artifact. Baseline uses the latest successful main measurement shown below.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 20.9 KiB 20.9 KiB 0 B (0.0%) 29.3 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.2 KiB 21.2 KiB 0 B (0.0%) 29.3 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: de96f28 · PR result: 3552009 · Source CI: failure

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@alexcstark
alexcstark merged commit ecbb4ef into main Oct 11, 2026
23 of 25 checks passed
@alexcstark
alexcstark deleted the fix/sqlite-synchronous-normal branch October 11, 2026 22:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XS vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant