Daemon becomes unresponsive in native HTTP/2 outgoing-buffer code on Node 26.7.0

[Zigoljube]

Summary

A DKG 10.0.18 daemon became persistently unresponsive during bounded archive ingestion. A native sample captured the main thread in Node HTTP/2 outgoing-buffer management with a very large process footprint. Oxigraph remained responsive.

This is a runtime investigation, not yet proof of a defect in DKG application code. Node/Undici/nghttp2 versus DKG transport/error handling has not been isolated.

Environment

• DKG CLI/daemon 10.0.18 (npm installation)
• Node.js 26.7.0
• macOS 26.2 ARM64
• Base mainnet (8453), existing edge node
• Managed external Oxigraph 0.5.8 (oxigraph-server)
• Normal supervised daemon launch; existing identity and storage preserved

Observed reproduction sequence

This is a live reproduction, not a minimal deterministic reproducer. Do not reproduce on an unprotected production node: the observed footprint was large.

1. Start the existing daemon after a cold backup and cleanup of its positively identified orphaned managed store.
2. Healthy API checks and an idle profiling window complete without HTTP failures.
3. Run sequential bounded archive ingestion through authenticated knowledge-asset create/write/share APIs, with read-back verification and durable checkpoints. No VM publishing.
4. Eight repair batch checkpoints reach verified-shared. During a later share/read-back sequence, the daemon becomes unresponsive.

Temporal overlap with sharing does not identify the failing HTTP/2 connection or prove sharing itself is the cause.

Evidence: 26 September 2026, UTC

• Approximately 100% worker CPU; local /api/status times out.
• Direct store probe ASK {} succeeds in 0.004 seconds while the API is unresponsive.
• At 07:55:21.886 the daemon logs:

[fatal] Uncaught exception: InformationalError: socket idle timeout
    at Timeout.onHttp2SessionIdleTimeout [as _onTimeout] (node:internal/deps/undici/undici:8713:19)
    at listOnTimeout (node:internal/timers:687:11)
    at process.processTimers (node:internal/timers:618:7)

A two-second native macOS sample at 07:56:48 captured all 1,069 main-thread samples through:

node::Environment::RunAndClearNativeImmediates
  -> node::http2::Http2Session::MaybeScheduleWrite callback
  -> node::http2::Http2Session::SendPendingData
  -> node::http2::Http2Session::CopyDataIntoOutgoing
  -> node::http2::Http2Session::PushOutgoingBuffer

Of those, 1,027 samples continued into:

std::vector<node::http2::NgHttp2StreamWrite>::__swap_out_circular_buffer

The sample reports Physical footprint: 44.3G, peak 45.1G. This is macOS process footprint, not resident RAM or measured JavaScript heap size. The stack and footprint suggest native buffering/memory-growth pathology, but do not establish the precise allocation mechanism.

The inspector-based replay profiler never received Profiler.stop before daemon termination, so no completed replay JavaScript CPU profile is available. The successful earlier idle profile is not the stall profile.

Stored outcome versus client timeout

The final batch timed out from the client's perspective. Subsequent direct scoped store verification found all 134 intended triples in Shared Memory, zero missing. Daemon lifecycle reconciliation remained pending. A timeout must not be treated as proof that a mutation failed or retried blindly.

Recovery impact

At 07:57:24.488 the supervisor killed the worker after five failed liveness probes. Managed Oxigraph survived with the database lock, and replacements failed to start. This separately tracked recovery defect is #1666; additional reproduction evidence will be posted there.

Expected behavior / investigation

• Identify the HTTP/2 connection and error path responsible for the native stall.
• Compare the same workload on a supported LTS Node runtime without changing identity/storage.
• Establish whether Node/Undici/nghttp2 fixes or DKG transport/error handling changes are needed.
• Preserve safe reconciliation of uncertain write/share outcomes.

Limits and related work

• Not yet reproduced on LTS Node.
• DKG 10.0.19 has not been tested for this failure; no claim that upgrading fixes it.
• No claim that RPC latency alone caused the stall.
• Investigate recurring main-thread pauses across mainnet cores, including 10.0.19 #2812 concerns shorter recurring pauses on other nodes. This is adjacent but distinct: sustained unresponsiveness, captured native HTTP/2 stack, and large footprint. A shared root cause is not established.
• Only sanitized excerpts are included: no credentials, private graph identifiers, machine paths, raw logs or source payloads are attached.

[branarakic-agent]

Likely trigger: Node 26 bundles undici 8, whose fetch negotiates HTTP/2 with any TLS server that offers it; see undici's v7 to v8 migration notes. Node 22 and 24 bundle undici 6 and 7, which stay on HTTP/1.1. The daemon's chain JSON-RPC calls go through the global fetch, so on Node 26 every RPC call runs over HTTP/2. Your stack is inside Node's HTTP/2 session code, and the uncaught InformationalError: socket idle timeout comes from undici's HTTP/2 idle timer. DKG does not enable HTTP/2 itself. CI, devnet and the release harness all run Node 22, so this path was untested.

Mitigation for 10.0.20: #2830 keeps both chain RPC fetch sites on HTTP/1.1 where Node's undici would negotiate HTTP/2. It does this with an HTTP/1.1-only undici 7 dispatcher, the same pattern the local-agent forwards use, and changes nothing on Node 22 and 24. It is a mitigation, not a proven root-cause fix: it removes the daemon's heavy HTTPS traffic from the HTTP/2 path your stack points at.

Until then: run the daemon on Node 22 LTS, the line every release is tested on, or on Node 24.

Your recovery defect after the worker kill, an orphaned Oxigraph holding the LOCK, is #1666, and #2775 fixes it in 10.0.20.

🤖 Generated with Claude Code

[branarakic-agent]

Update: #2830 is merged into testnet-canary (as 092e27676) and ships in 10.0.20.

• On Node 26: every chain JSON-RPC call now asks fetch's own dispatcher to refuse HTTP/2. Proxies set through NODE_USE_ENV_PROXY and dispatchers installed with setGlobalDispatcher still carry the calls.
• Tested: real fetches on Node 26.7.0 against a server that offers both protocols stayed on HTTP/1.1 in every case: with Node's default dispatcher, through NODE_USE_ENV_PROXY and through a CONNECT proxy. A new CI lane runs these checks on Node 26.
• On Node 22 and 24: nothing changes.
• Other undici versions: the daemon logs [chain] RPC calls may use HTTP/2 (#2828) once instead of guessing.

What it doesn't cover: other outgoing HTTPS requests on Node 26 still use Node's defaults (auto-update checks, the integrations registry, optional LLM calls). They are low volume.

So this removes the daemon's heavy HTTP/2 traffic, but it doesn't prove the stall's root cause. Until 10.0.20 is out, running on Node 22 or 24 remains the safe choice. I'm leaving this issue open until 10.0.20 has run on Node 26 without a recurrence.