Skip to content

Network sidecar crashes on reconnect: SidecarConfig required for NetworkNodeAdapter.start() #17

Description

@Hafuyya

Summary

When HYPERSPACE_NETWORK_SIDECAR=1 (default), any network disconnect triggers a reconnect path that crashes with:

Fatal error: Error: SidecarConfig required for NetworkNodeAdapter.start()

This forces a full process restart every time the sidecar loses connectivity. The only workaround is HYPERSPACE_NETWORK_SIDECAR=0 (in-process networking), which introduces a separate problem: training gradient uploads block the libp2p event loop, causing heartbeat timeouts and ~10-minute crash cycles.

Root Cause

The CLI has two this.node.start() calls:

  1. Initial start (in try/catch) — passes sidecarConfig
  2. Reconnect path — calls await this.node.start() without sidecarConfig

NetworkNodeAdapter.start() requires sidecarConfig to spawn hyperspace-network, so the second call always throws.

Reproduction

  1. Start with HYPERSPACE_NETWORK_SIDECAR=1 (default)
  2. Wait for any network disconnect (peer timeout, transient connectivity loss, etc.)
  3. Observe crash: SidecarConfig required for NetworkNodeAdapter.start()
  4. Service restarts, works until next disconnect, crashes again

Tested on CLI versions v5.37.3 through v5.39.6 — all affected.

Impact

  • With sidecar (default): Node crash-loops on any disconnect event
  • Without sidecar (workaround): Training gradient upload + FedAvg aggregation blocks the single-threaded event loop, causing GossipSub heartbeat timeouts → peer disconnections → node restart every ~10 minutes

This means there is currently no stable configuration for miner nodes that participate in training. The sidecar fix would resolve both issues since networking would run in a separate process, unaffected by training I/O.

Environment

  • OS: Ubuntu 24.04, Linux 6.17.0
  • GPU: RTX 5090 32GB
  • CLI: v5.39.2 (also tested v5.37.3, v5.37.4, v5.37.6, v5.39.6)
  • Chain: v1.7.4
  • Role: miner

Suggested Fix

Pass sidecarConfig to the reconnect start() call, or cache it from the initial start so NetworkNodeAdapter can reuse it on reconnect.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions