container_state/container_log/stream endpoints return 500 (Bugsnag noise) when container execution not yet available

[morgan-wowk]

Summary

RuntimeError: Execution with id='...' does not have container execution information.
surfaces in Bugsnag as an uncaught 500. This is an expected transient state (execution
PENDING, pod/container not yet created), not a server fault. Should return a non-error
HTTP status and not notify Bugsnag.

Root cause

An ExecutionNode row exists before its ContainerExecution row. The orchestrator creates
and links the ContainerExecution only after launch (orchestrator_sql.py: created ~L642,
linked execution.container_execution = container_execution ~L674). Meanwhile
ExecutionNode.container_execution_status can already be PENDING. Window:
status == PENDING but execution.container_execution is None.

Three endpoints raise in that window (cloud_pipelines_backend/api_server_sql.py):

• GET /api/executions/{id}/container_state       → get_container_execution_state (~L765) → RuntimeError
• GET /api/executions/{id}/container_log          → get_container_execution_log (~L894) → RuntimeError
• GET /api/executions/{id}/stream_container_log   → stream_container_execution_log (~L965) → ApiServiceError

api_router.py has handlers only for ItemNotFoundError→404, PermissionError→403,
ItemAlreadyExistsError→409, ApiValidationError→422, NotImplementedError→501. RuntimeError/
ApiServiceError are uncaught → FastAPI default 500 → Bugsnag.

Impact

Recurring Bugsnag noise. Execution stuck in PENDING (no pod, various causes) → repeated
500s as the UI polls. Real "execution does not exist" 404s buried under expected-state noise.

Suggested solution

1. New exception e.g. ContainerExecutionNotReadyError in errors.py, distinct from
      ItemNotFoundError (which means the execution itself is absent).
2. Raise it at all three sites when the ExecutionNode exists but container_execution is None
      and status is pre-launch/pending. Unify the current RuntimeError + ApiServiceError to it.
3. Add an api_router.py handler → 409 Conflict (see below). Machine-readable body e.g.
      {"reason": "container_not_ready", "execution_status": ""}. Do NOT notify Bugsnag.[8:31 AM]## HTTP status recommendation

• 409 Conflict (recommended): resource exists, current state (PENDING, no pod) conflicts
    with the request; retryable; distinguishes from 404 (execution absent) and 500 (fault).
• 404: acceptable but weaker — conflates with "execution does not exist" (already 404) and
    "wrong URL"; client can't tell "not yet" from "never".
• 200 with explicit state (for /container_state only): return status: PENDING with container
    fields null. Needs a nullable response-schema change; richest long-term answer for the poll loop.

Precedent: get_container_execution_log already returns a graceful response (not raise) for
SYSTEM_ERROR with no container row (~L878). Extend that "no row is a valid answer" pattern.

Acceptance

• container_state/container_log/stream for a PENDING execution with no container row → 409
    (or chosen code), not 500.
• No Bugsnag event on that path.
• Genuine "execution does not exist" still returns 404