Summary
When HYPERSPACE_NETWORK_SIDECAR=1 (default), any network disconnect triggers a reconnect path that crashes with:
Fatal error: Error: SidecarConfig required for NetworkNodeAdapter.start()
This forces a full process restart every time the sidecar loses connectivity. The only workaround is HYPERSPACE_NETWORK_SIDECAR=0 (in-process networking), which introduces a separate problem: training gradient uploads block the libp2p event loop, causing heartbeat timeouts and ~10-minute crash cycles.
Root Cause
The CLI has two this.node.start() calls:
- Initial start (in try/catch) — passes
sidecarConfig ✅
- Reconnect path — calls
await this.node.start() without sidecarConfig ❌
NetworkNodeAdapter.start() requires sidecarConfig to spawn hyperspace-network, so the second call always throws.
Reproduction
- Start with
HYPERSPACE_NETWORK_SIDECAR=1 (default)
- Wait for any network disconnect (peer timeout, transient connectivity loss, etc.)
- Observe crash:
SidecarConfig required for NetworkNodeAdapter.start()
- Service restarts, works until next disconnect, crashes again
Tested on CLI versions v5.37.3 through v5.39.6 — all affected.
Impact
- With sidecar (default): Node crash-loops on any disconnect event
- Without sidecar (workaround): Training gradient upload + FedAvg aggregation blocks the single-threaded event loop, causing GossipSub heartbeat timeouts → peer disconnections → node restart every ~10 minutes
This means there is currently no stable configuration for miner nodes that participate in training. The sidecar fix would resolve both issues since networking would run in a separate process, unaffected by training I/O.
Environment
- OS: Ubuntu 24.04, Linux 6.17.0
- GPU: RTX 5090 32GB
- CLI: v5.39.2 (also tested v5.37.3, v5.37.4, v5.37.6, v5.39.6)
- Chain: v1.7.4
- Role: miner
Suggested Fix
Pass sidecarConfig to the reconnect start() call, or cache it from the initial start so NetworkNodeAdapter can reuse it on reconnect.
Summary
When
HYPERSPACE_NETWORK_SIDECAR=1(default), any network disconnect triggers a reconnect path that crashes with:This forces a full process restart every time the sidecar loses connectivity. The only workaround is
HYPERSPACE_NETWORK_SIDECAR=0(in-process networking), which introduces a separate problem: training gradient uploads block the libp2p event loop, causing heartbeat timeouts and ~10-minute crash cycles.Root Cause
The CLI has two
this.node.start()calls:sidecarConfig✅await this.node.start()withoutsidecarConfig❌NetworkNodeAdapter.start()requiressidecarConfigto spawnhyperspace-network, so the second call always throws.Reproduction
HYPERSPACE_NETWORK_SIDECAR=1(default)SidecarConfig required for NetworkNodeAdapter.start()Tested on CLI versions v5.37.3 through v5.39.6 — all affected.
Impact
This means there is currently no stable configuration for miner nodes that participate in training. The sidecar fix would resolve both issues since networking would run in a separate process, unaffected by training I/O.
Environment
Suggested Fix
Pass
sidecarConfigto the reconnectstart()call, or cache it from the initial start soNetworkNodeAdaptercan reuse it on reconnect.