Observer role registration either fails or is incomplete when multiple runtimes are enabled

[gw0]

SUMMARY

A race condition happens when registering an observer node with multiple runtimes enabled on Testnet. It sometimes fails with node update not allowed (removal of runtimes is not allowed), sometimes registers with an empty list (should not happen), and sometimes only registers for Emerald Testnet runtime (the only non-TEE). Restarting the node still leaves it in one of the failed states. Unless something else is missing, after the next epoch transition (when the node registration is retried) it successfully re-registers for all runtimes.

It seems the registration process is separate from runtime readiness checks. It registers as only an observer node for Emerald Testnet, because TEE runtimes are not yet ready, but once they become ready it does not re-register. This might also explain why it sometimes complains with node update not allowed (because other runtimes are not yet running and it attempts to register only for Emerald Testnet what represents a downgrade).

ISSUE TYPE

• Bug Report

COMPONENT NAME

worker/registration

OASIS NODE VERSION

24.3.1

OS / ENVIRONMENT

Ubuntu 22.04

STEPS TO REPRODUCE

1. Set up a ParaTime Client node to register as an observer role for all runtimes on Testnet (needs whitelisting)

mode: client
registration:
  entity_id: ...
runtime:
  paths:
    - /node/runtimes/cipher-testnet.orc
    - /node/runtimes/emerald-testnet.orc
    - /node/runtimes/sapphire-testnet.orc
    - /node/runtimes/pontusx-testnet.orc
    - /node/runtimes/pontusx-devnet.orc

2. Check registration status in control status output, you get one of:

  "registration": {
    "last_attempt_successful": false,
    "last_attempt_error_message": "registry: node update not allowed",
## or:
  "registration": {
    "last_attempt_successful": true,
    "descriptor": {
      "runtimes": null,
      "roles": "observer",
## or:
  "registration": {
    "last_attempt_successful": true,
    "descriptor": {
      "runtimes": [
        { "id": "00000000000000000000000000000000000000000000000072c8215e60d5bca7",... }
      ],
      "roles": "observer",

Right after it successfully registers for Emerald Testnet, the status of other runtimes is:

    "status": "ready",
        "status": "waiting for hosted runtime provision",
        "status": "syncing rounds",
        "status": "waiting for hosted runtime provision",
        "status": "syncing rounds",
        "status": "waiting for hosted runtime provision",
        "status": "syncing rounds",
        "status": "ready",
        "status": "syncing rounds",
        "status": "waiting for hosted runtime provision",
        "status": "syncing rounds",

EXPECTED RESULTS

Successful registration as observer node for all runtimes.

[kostko]

Probably the issue here is that client runtimes do not block registration, as usually registration should not happen until all runtimes are ready (this is e.g. the case for executor nodes).

[kostko]

Yes, currently the problem is that for client workers registration immediately becomes unblocked instead of waiting for readiness:

oasis-core/go/worker/client/worker.go

Line 141 in 60cb98e

rp.SetAvailable(commonNode.RegisterNodeRuntime)

Should be easy to fix.