Description
When Charon starts before its configured beacon node REST API is available (common with split hosts / LXC / Proxmox boot order), Charon can print one or two startup log lines (e.g. lock file loaded) and then go silent. The process remains running (systemctl shows active), but it never becomes operational.
Importantly: once the beacon node later becomes healthy, Charon does not recover. Killing/restarting Charon after the BN is up fixes it immediately.
This is worse than “BN unreachable while Charon keeps retrying,” because the process stays wedged and systemd Restart=on-failure never fires.
Environment
Charon installed via systemd (EthPillar), not Docker Compose
Topology: Charon + VC on one host/LXC; beacon node on a separate host/LXC
BN client: not Nimbus (so not the json_requests case)
After host reboot, BN CT and Charon CT can start in parallel / Charon can win the race
Workaround that works: restart Charon (or reboot the Charon CT) after BN REST is answering
Reproduction
Configure Charon with --beacon-node-endpoints=http://:5052
Ensure BN REST is down / not listening yet
Start Charon
Observe: few log lines, then silence; process stays up
Bring BN REST up and confirm it answers (/eth/v1/node/version)
Observe: Charon still stuck; VC still cannot usefully talk to Charon :3600
systemctl restart charon → Charon starts normally
Expected behavior
Charon should either:
Retry BN-dependent startup (genesis/spec/etc.) with backoff until the BN is reachable, then continue wiring and start the validator API; or
Fail fast and exit so the process supervisor can restart it once the BN is up
It should not remain permanently stuck in a half-started state after the BN recovers.
Suspected cause (from reading current main)
After "Lock file loaded", startup still synchronously depends on the BN before life.Run starts the validator API:
sse.StartListener → FetchGenesisTime / FetchSlotsConfig
wireCoreWorkflow → more BN fetches (NewDutyDeadlineFunc, NewDutyGater, fork/slot config)
Only then is the validator API (:3600) registered/started
Charon constructs the eth2 client with WithAllowDelayedStart(true), but then immediately requires successful BN calls during startup. If that path hangs or gets stuck without exiting, the process looks “active” while unhealthy, and delayed start does not help once the BN appears later.
Relevant areas:
app/app.go — order after lock load: newETH2Client → sse.StartListener → wireCoreWorkflow → life.Run
app/sse/listener.go — StartListener blocks on BN genesis/spec
app/eth2wrap/eth2wrap.go — WithAllowDelayedStart(true) on eth2http.New
Impact
Split BN / Charon deployments (separate VMs/LXCs) are unreliable across host reboot
Operators see VC logs complaining that Charon’s BN API is unavailable
Manual restart is required even after the BN is healthy
Systemd restart policy cannot heal a non-exiting hang
Description
When Charon starts before its configured beacon node REST API is available (common with split hosts / LXC / Proxmox boot order), Charon can print one or two startup log lines (e.g. lock file loaded) and then go silent. The process remains running (systemctl shows active), but it never becomes operational.
Importantly: once the beacon node later becomes healthy, Charon does not recover. Killing/restarting Charon after the BN is up fixes it immediately.
This is worse than “BN unreachable while Charon keeps retrying,” because the process stays wedged and systemd Restart=on-failure never fires.
Environment
Charon installed via systemd (EthPillar), not Docker Compose
Topology: Charon + VC on one host/LXC; beacon node on a separate host/LXC
BN client: not Nimbus (so not the json_requests case)
After host reboot, BN CT and Charon CT can start in parallel / Charon can win the race
Workaround that works: restart Charon (or reboot the Charon CT) after BN REST is answering
Reproduction
Configure Charon with --beacon-node-endpoints=http://:5052
Ensure BN REST is down / not listening yet
Start Charon
Observe: few log lines, then silence; process stays up
Bring BN REST up and confirm it answers (/eth/v1/node/version)
Observe: Charon still stuck; VC still cannot usefully talk to Charon :3600
systemctl restart charon → Charon starts normally
Expected behavior
Charon should either:
Retry BN-dependent startup (genesis/spec/etc.) with backoff until the BN is reachable, then continue wiring and start the validator API; or
Fail fast and exit so the process supervisor can restart it once the BN is up
It should not remain permanently stuck in a half-started state after the BN recovers.
Suspected cause (from reading current main)
After "Lock file loaded", startup still synchronously depends on the BN before life.Run starts the validator API:
sse.StartListener → FetchGenesisTime / FetchSlotsConfig
wireCoreWorkflow → more BN fetches (NewDutyDeadlineFunc, NewDutyGater, fork/slot config)
Only then is the validator API (:3600) registered/started
Charon constructs the eth2 client with WithAllowDelayedStart(true), but then immediately requires successful BN calls during startup. If that path hangs or gets stuck without exiting, the process looks “active” while unhealthy, and delayed start does not help once the BN appears later.
Relevant areas:
app/app.go — order after lock load: newETH2Client → sse.StartListener → wireCoreWorkflow → life.Run
app/sse/listener.go — StartListener blocks on BN genesis/spec
app/eth2wrap/eth2wrap.go — WithAllowDelayedStart(true) on eth2http.New
Impact
Split BN / Charon deployments (separate VMs/LXCs) are unreliable across host reboot
Operators see VC logs complaining that Charon’s BN API is unavailable
Manual restart is required even after the BN is healthy
Systemd restart policy cannot heal a non-exiting hang