Skip to content

Add ELECTRIC_TCP_READ_TIMEOUT for the HTTP keep-alive idle timeout#4716

Open
alco wants to merge 3 commits into
mainfrom
alco--http-keepalive-read-timeout
Open

Add ELECTRIC_TCP_READ_TIMEOUT for the HTTP keep-alive idle timeout#4716
alco wants to merge 3 commits into
mainfrom
alco--http-keepalive-read-timeout

Conversation

@alco

@alco alco commented Jul 15, 2026

Copy link
Copy Markdown
Member

What

Adds ELECTRIC_TCP_READ_TIMEOUT to configure ThousandIsland's read_timeout (default 60s), which doubles as Bandit's HTTP keep-alive idle timeout: connections that have waited this long for their next request are closed. The option mirrors ELECTRIC_TCP_SEND_TIMEOUT (human-readable duration) and threads through tcp_read_timeoutthousand_island_options/1. Default is unchanged (nil → ThousandIsland's 60s).

Includes a docs entry in website/docs/sync/api/config.md and a changeset.

Why

When Electric runs behind a connection-pooling proxy, the target's keep-alive timeout must exceed the proxy's idle timeout. If it doesn't, the proxy can reuse a pooled connection at the same moment Bandit's idle timer closes it (the close is a bare FIN — nothing announces it in-band), and the resulting TCP reset surfaces to clients as an intermittent 502 Bad Gateway.

Expose ThousandIsland read_timeout (default 60s), which doubles as the
keep-alive idle timeout: Bandit reaps connections idle longer than this
without announcing the close via connection: close. Behind a pooling
proxy (e.g. AWS ALB) the value must exceed the proxy idle timeout, or
the proxy races the unannounced close when reusing a pooled connection
and clients see intermittent 502s. Cloud diagnosis: daily-recurring
storms of ALB-minted 502s (~709k on Jul 9) from mass idle reaping of
cold connection pools.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEErBr3SXEa5VprZu26ZkP
@alco alco added the claude label Jul 15, 2026
@netlify

netlify Bot commented Jul 15, 2026

Copy link
Copy Markdown

Deploy Preview for electric-next ready!

Name Link
🔨 Latest commit 57dc7e4
🔍 Latest deploy log https://app.netlify.com/projects/electric-next/deploys/6a588d053b3bbd0008154251
😎 Deploy Preview https://deploy-preview-4716--electric-next.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

@claude

claude Bot commented Jul 15, 2026

Copy link
Copy Markdown

Claude Code Review

Summary

Adds ELECTRIC_TCP_READ_TIMEOUT to configure ThousandIsland's read_timeout (the HTTP keep-alive idle timeout), threading it through runtime.exsElectric.Config defaults → thousand_island_options/1. This iteration is a cleanup pass (commit 57dc7e4a0 "human touch") that trims two explanatory comments and simplifies one doc sentence. The functional change is unchanged from iteration 2, remains correct, and is ready to merge.

What's Working Well

  • Core logic and test coverage are untouched and still correct. read_timeout_opts is still appended to the top-level keyword list (acceptor_opts ++ read_timeout_opts ++ transport_opts ++ genserver_opts), not nested under transport_options where it would silently no-op. The config_test.exs case still asserts both the top-level mapping (thousand_island_options[:read_timeout] == 180_000) and that the default omits the key, so the placement stays locked in even though the inline comment explaining why was removed.
  • Consistent with existing conventions. Mirrors ELECTRIC_TCP_SEND_TIMEOUT exactly — same parse_human_readable_time!/1 parser, same case get_env(...) shape, same @defaults entry.
  • Backwards compatible. Default remains nil[], preserving ThousandIsland's built-in default; no behavior change for existing deployments.

Issues Found

Critical (Must Fix)

None.

Important (Should Fix)

None.

Suggestions (Nice to Have)

  • Doc wording now re-couples the default to ThousandIsland's without attribution. config.md changed from "When unset, the HTTP server's default applies (60 seconds, ThousandIsland's current default)" back to "Defaults to 60 seconds." Since the config default is actually nil (the key is omitted and ThousandIsland supplies the 60s), the docs will silently drift if that upstream default ever changes. This is accurate today and was a deliberate human edit, so it's purely informational — no action needed. The same low-stakes note applies to the two removed inline comments (in application.ex and the test): the code is still correct without them, just less self-documenting about the top-level-vs-transport subtlety.

Issue Conformance

No linked issue. Per project conventions PRs should reference the issue/incident they address, but the PR description is unusually thorough — it documents the production incident (ALB 502 storm), the root cause (keep-alive idle timeout inverted relative to the proxy idle timeout), and the remediation. The changeset is present and correctly targets @core/sync-service (patch). Monorepo impact is limited to the sync-service; no client-facing API or HTTP contract change.

Previous Review Status

Both iteration-1 suggestions were addressed in iteration 2 (01cbcaa1c), and I flagged no outstanding issues in iteration 2. The new commit 57dc7e4a0 is a cosmetic cleanup only:

  • Removed the read_timeout keep-alive explanatory comment in application.ex.
  • Removed the "top-level option, nesting would no-op" comment in config_test.exs (the assertion it explained still stands).
  • Simplified the config.md default phrasing to "Defaults to 60 seconds."

No functional regression. Nothing from prior reviews has regressed.


Review iteration: 3 | 2026-07-16

@codecov

codecov Bot commented Jul 15, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.05%. Comparing base (4051e0f) to head (57dc7e4).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4716   +/-   ##
=======================================
  Coverage   60.04%   60.05%           
=======================================
  Files         397      397           
  Lines       43766    43766           
  Branches    12586    12589    +3     
=======================================
+ Hits        26281    26284    +3     
+ Misses      17403    17401    -2     
+ Partials       82       81    -1     
Flag Coverage Δ
packages/agents 72.64% <ø> (ø)
packages/agents-mcp 77.70% <ø> (ø)
packages/agents-mobile 80.67% <ø> (ø)
packages/agents-runtime 83.73% <ø> (+0.01%) ⬆️
packages/agents-server 75.67% <ø> (ø)
packages/agents-server-ui 8.32% <ø> (ø)
packages/electric-ax 51.06% <ø> (ø)
packages/experimental 87.73% <ø> (ø)
packages/react-hooks 86.48% <ø> (ø)
packages/start 82.83% <ø> (ø)
packages/typescript-client 91.89% <ø> (+0.11%) ⬆️
packages/y-electric 56.05% <ø> (ø)
typescript 60.05% <ø> (+<0.01%) ⬆️
unit-tests 60.05% <ø> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

alco and others added 2 commits July 15, 2026 13:44
Test asserts tcp_read_timeout maps to a top-level ThousandIsland option
(nesting it in transport_options would silently no-op), and that nil
omits it. Docs now attribute the 60s default to ThousandIsland rather
than presenting it as our own.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AEErBr3SXEa5VprZu26ZkP

@robacourt robacourt left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants