feat: depends_on should be wait-until-ready, not just start-order

[mighty840]

Problem

`depends_on = ["backend"]` in `service.toml` currently guarantees start-order (backend container is created + started before frontend), but not readiness — the backend can be in any state between "docker started" and "actually serving traffic" when the frontend boots.

That's fine for most services that themselves retry, but a large class of container entrypoints does one-shot DNS resolution at boot (nginx configs with `proxy_pass http://backend:8000\`, envoy configs with static clusters, some database driver connect-strings) — a boot-time TCP or DNS failure kills the process and, combined with `restart=no` (see orca#121), leaves the service dead.

Compose has this same problem and papered over it with `depends_on: condition: service_healthy` requiring the depended-on service to have a healthcheck. Kubernetes solved it with readiness probes gating init containers.

Repro

Real incident 2026-07-07, cluster `breakpilot`:

1. Something triggers a mass container recreate on the agent `ubuntu-16gb-fsn1-1` (in our case it was orca#120's reconcile scope bleed).

2. Orca starts `erp-backend` at 07:46:17 UTC. Docker registers its network alias asynchronously.

3. Orca starts `erp-frontend` at 07:46:16 UTC (per `depends_on = ["erp-backend"]`).

   Note the second: frontend started 1 second BEFORE backend. `depends_on` was violated. In our case both were on the same agent, launched by the same orca-agent process, and yet the ordering wasn't enforced.

4. nginx in erp-frontend does a config parse; `proxy_pass http://erp-backend:8000\` → `getaddrinfo("erp-backend")` → `[emerg] host not found in upstream`. Process exits.

5. Container marked `restart=no`. Service dead.

Even if orca had honored strict start-order, the DNS alias would have taken another few hundred ms to propagate through Docker's embedded DNS. That's a race, not a start-order fix.

Proposed API

```toml
[[service]]
name = "erp-frontend"
depends_on = ["erp-backend"] # existing: start-order-ish
wait_for = [ # NEW: block start until reachable
"erp-backend:tcp:8000", # TCP connect
"erp-db:tcp:3306:30s", # with timeout
"erp-backend:http:/health", # HTTP 200
]
```

Or richer:

```toml
[[service.wait_for]]
target = "erp-backend"
check = "tcp"
port = 8000
timeout = "30s"

[[service.wait_for]]
target = "erp-db"
check = "http"
path = "/health"
port = 8000
expect_status = 200
```

Semantics: orca launches the container only after every `wait_for` check passes. Implementation-wise this can be a tiny sidecar (`orca-wait`) that runs before the real entrypoint, or a wrapper around the entrypoint command orca injects, or a real orca-agent-side pre-flight check.

Compat: `depends_on` keeps its current start-order semantic. `wait_for` is additive.

Alternatives considered

• Wrap every entrypoint with a wait-loop. What we did as a workaround (`sh -c 'until getent hosts backend; do sleep 1; done; exec entrypoint'`). Works, but every service operator has to remember to do it, and the loop needs to be written per-service depending on what constitutes "ready" (DNS resolvable? TCP open? HTTP 200 on /health?).
• Rely on the restart_policy from orca#121. Restart-on-failure fixes many symptoms but doesn't address the actual race — a service can crash-loop indefinitely if the retry timing collides with the backend's readiness window. Wait-for-ready is the semantic fix; restart-policy is the safety net.
• Kubernetes-style pod readinessProbes. Would need to model probe results in orca's status API. Bigger change than `wait_for` at the depends_on level.

Trade-offs

• Adds a small runtime dependency (either an injected wrapper or an orca-agent pre-flight check).
• If a `wait_for` target never becomes ready, the depending container never starts. Orca needs a clear "stuck waiting" state distinct from "crashed" in `orca status`.
• Circular waits should be a validation error at deploy time.

Related

• orca#120 — reconcile scope bleed (the trigger that exposed this).
• orca#121 — restart-policy support (the safety net that would mask this from becoming outages).
• orca#119 — cluster-native status page (would detect these earlier).

Together, #119 + #120 + #121 + this issue = the reconcile/startup hardening pass.

---

🤖 Filed via Claude Code