feat: SLS-494 sdk supervises worker initialization code

[jasonwang-runpod]

Problem

Currently, workers do their startup work (model load, engine start) before calling runpod.serverless.start(), so the SDK never sees it. When that work hangs or crashes, the worker never takes a request - so the request is stuck in IN_QUEUE with no obvious error until its TTL expires, and the reason is buried in worker logs. Users read this as "Runpod lost my request".

Solution

Enable the SDK to supervise initialization code. start() accepts an optional initializer callable and init_timeout.

• The worker runs the initializer as a fourth concurrent task, next to job take, job run, and job stop.
• Job take keeps running, so a request still moves out of the queue immediately.
• Handler execution waits until initialization finishes.
• On failure or timeout, the worker fails the request it is holding through the existing /job-done route, with the reason and the tail of the initializer's stdout/stderr, then exits so the platform respawns it under existing backoff.
• Handler failures now carry captured stdout/stderr too, which is usually where the real cause is (CUDA errors, OOM kills).

Every reported field is clipped so a large message or log shouldn't result in a large job-done body.

Out of scope: Container failures (image pull, host OOM) have no request in hand and no SDK, so they cannot be reported per request.

Screenshots

Requests now fail fast and store init failure logs

Logs are still present in their own UI

Testing

uv run pytest: 674 passed.

Ran real serverless endpoint tests on RTX 4090, with the SDK built from this branch. Validated that both init errors and handler errors now store failure logs on the request itself.

Case	Result
initializer succeeds	COMPLETED; a request was taken, held, and ran against an initialized worker
initializer raises after 3s	FAILED with init_failed + RuntimeError + CUDA OOM text + traceback + stdout/stderr
async initializer raises immediately	FAILED - the worker held no request yet, so it claimed one and failed it. Without doing this, the request waits out its TTL.
initializer hangs, init_timeout=20	FAILED with InitializerTimeout, initializer exceeded init_timeout of 20s
initializer hangs, no init_timeout	request goes IN_PROGRESS and is stuck until the platform's executionTimeout exceeded. Omitting the timeout opts out of any SDK bound, by design.
CUDA fault during init	init_failed, classified as cuda_error
concurrency 3, init fails with 3 requests in hand	all three FAILED with the same reason within a second; none silently requeued
handler raises (no initializer)	FAILED with the usual JSON error, now includes the stdout and stderr
generator handler raises mid-stream (no initializer)	FAILED after one chunk; error keeps its original plain string shape with logs appended

[Copilot]

Pull request overview

This PR introduces a supervised startup phase for serverless workers by adding an optional initializer callable and init_timeout to the serverless worker config, ensuring model/engine startup failures (or hangs) are surfaced quickly and the worker exits before taking jobs.

Changes:

• Added rp_initializer module to run an initializer with timeout handling, structured init_failed logging, and best-effort platform reporting.
• Updated the worker startup sequence to run initialization before starting the job loop.
• Added unit tests covering sync/async initializers, error wrapping, timeouts, reporting, and job-loop gating.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
tests/test_serverless/test_initializer.py	Adds unit tests for initializer execution, failure/timeout behavior, reporting, and gating before the job loop.
runpod/serverless/worker.py	Runs supervised initialization before starting the JobScaler/job loop.
runpod/serverless/modules/rp_initializer.py	Implements supervised initializer execution, timeout/error wrapping, structured logging, and best-effort reporting.
runpod/serverless/init.py	Documents new initializer and init_timeout config options in the serverless start() docstring.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[jasonwang-runpod]

bugbot run

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (4)

runpod/serverless/modules/rp_job.py:347

• run_job_generator bounds the handler/traceback string via clip(...), but then appends up to 16KB of captured logs afterward, which can defeat the intended bounding and inflate the streamed error payload. Consider clipping the final assembled error string instead.

            error = clip(f"handler: {str(err)} \ntraceback: {traceback.format_exc()}")
            if captured_logs:
                error += f"\nlogs:\n{captured_logs}"
            yield {"error": error}

runpod/serverless/modules/rp_initializer.py:96

• Docstring typo: “abandoned an die” should be “abandoned and die”.

    """Run a blocking callable on a daemon thread instead of `asyncio.to_thread` so if stuck,
    it can be abandoned an die without blocking the executor shutdown and process exit."""

runpod/serverless/modules/rp_initializer.py:140

• run_initializer_async treats timeout=0 as “no timeout” because it uses a truthiness check (if timeout:). If a user sets init_timeout: 0 expecting an immediate timeout, it will instead run unbounded.

        if timeout:
            await asyncio.wait_for(asyncio.ensure_future(awaitable), timeout=timeout)
        else:
            await awaitable

runpod/serverless/init.py:150

• The docstring says the initializer runs “before the job loop begins”, but the implementation can still acquire jobs while the initializer runs (the gate only prevents the handler from executing). This wording is misleading for SDK users.

    config["initializer"] (Callable, optional): Startup work before the job loop begins,
        e.g. loading a model or starting an inference engine.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

runpod/serverless/modules/rp_initializer.py:133

• An initializer that raises asyncio.TimeoutError itself is always converted to InitializerTimeout; with no configured deadline this even reports initializer exceeded init_timeout of Nones. Track the invocation task and only classify TimeoutError as an SDK deadline when wait_for actually cancelled that task, otherwise wrap the user's exception as InitializerError.

    except asyncio.TimeoutError as exc:
        raise InitializerTimeout(
            f"initializer exceeded init_timeout of {timeout}s"
        ) from exc

runpod/serverless/modules/rp_http.py:127

• HTTP 4xx/5xx responses are currently consumed as if the handler-started signal succeeded, so the final server error is neither raised nor logged (and eligible failures may not be retried as intended). Pass raise_for_status=True, matching _transmit, so the best-effort path still detects and logs rejected signals.

        async with retry_client.post(
            url,
            data="{}",
            headers={
                "charset": "utf-8",
                "Content-Type": "application/x-www-form-urlencoded",
            },
        ) as client_response:

[jhcipar]

my clanker tells me that sys.exit won't actually exit here if there are non-daemon threads still alive (e.g. ones vllm etc. spawn before init fails) — interpreter shutdown blocks joining them, so the worker can hang instead of exiting for respawn. probably should be os._exit(1) here like the fitness checks do (rp_fitness._terminate_unhealthy exists for exactly this reason)