Skip to content

discovery: poll for the meta instead of timing out + rich logging (0.6.3) - #21

Merged
flymysql merged 1 commit into
mainfrom
cursor/discovery-poll-logging-49da
Jun 2, 2026
Merged

discovery: poll for the meta instead of timing out + rich logging (0.6.3)#21
flymysql merged 1 commit into
mainfrom
cursor/discovery-poll-logging-49da

Conversation

@flymysql

@flymysql flymysql commented Jun 2, 2026

Copy link
Copy Markdown
Owner

Symptom

A node started before the meta crashed the whole host process:

Failed to create dynamic storage backend 'peercache': timed out
...
discovery.py register() → rpc.call("register") → TimeoutError: timed out

DiscoveryClient.start() did a single register() RPC and let a TimeoutError propagate, killing the SGLang scheduler.

Changes

  • Poll, don't time out. start() now calls _register_blocking() which retries register() indefinitely (until stop()), logging periodicallydiscovery: node=… waiting for meta … (attempt N, Ms elapsed): …; retrying in 2.0s. Once the meta is up it registers and continues. No more crash.
  • Operability logging (the requested visibility):
    • startup: node identity + endpoints, and which node is the meta (meta-host vs client);
    • meta logs every REGISTERED / re-register / DEREGISTERED / dead-node prune, each with the current member count + id list, and flags heartbeats from unknown nodes;
    • client logs successful registration (after N attempts, with member list), every heartbeat (known=, member count), and membership changes (joined=… left=…).
  • register_retry_interval knob (default 2.0s).

Verification

  • New test_register_polls_until_meta_is_up: a client started before the meta keeps polling (no exception) and registers once the DiscoveryServer comes up.
  • Full suite 53 passed.

Bumps to 0.6.3. Pure-Python (no extension rebuild needed).

Validate on the node (immediate)

pip install --force-reinstall --no-deps "git+https://github.com/flymysql/PeerCache@cursor/discovery-poll-logging-49da"

Start a worker before the meta — it now logs waiting for meta … and joins once the meta is up, instead of dying.

Open in Web Open in Cursor 

Registration no longer raises TimeoutError when the meta isn't up yet (which
crashed the SGLang scheduler); start() polls indefinitely with periodic
'waiting for meta ... retrying' logs and proceeds once it registers. Added
operability logging: startup identity + which node is the meta; meta logs each
register/re-register/deregister/prune with member count+list; client logs
successful registration, every heartbeat (known=, members), and membership
changes (joined/left). New test: register polls until the meta appears (no
crash). Full suite 53 passed.

Co-authored-by: 兰州小红鸡 <flyphp@outlook.com>
@flymysql
flymysql marked this pull request as ready for review June 2, 2026 06:13
@flymysql
flymysql merged commit 0218513 into main Jun 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants