Skip to content

Step 46: Warm-up — JIT pre-compilation, CUDA kernel pre-loading, configurable WarmupOptions #92

Closed

Description

@kkokosa

Summary

JIT pre-compilation pass at startup. CUDA kernel pre-loading. Configurable WarmupOptions. Readiness probe gates on warm-up completion.

At server startup, run configurable dummy inference passes after model loading but before marking IsReady=true. This triggers .NET Tier-0→Tier-1 JIT promotion (with Dynamic PGO profile data) and exercises CUDA/cuBLAS kernel pipelines, eliminating the cold-start latency penalty on the first real request.

Acceptance Criteria

  • WarmupOptions sealed record in DotLLM.Engine with Enabled, DummyPrompt, MaxTokens, Iterations properties
  • WarmupRunner static class that executes warm-up passes using TextGenerator
  • Server integration: warm-up runs in ServerStartup.LoadModel() before IsReady=true
  • Hot-swap: warm-up runs automatically on model swap via LoadModel()
  • CLI flags: --no-warmup (disable), --warmup-iterations N on serve command
  • Per-iteration timing logged with [dotllm] prefix
  • /ready probe returns 503 during warm-up, 200 after completion
  • docs/WARMUP.md documentation covering JIT, CUDA, Dynamic PGO, configuration
  • CLAUDE.md Documentation Index updated with WARMUP.md
  • docs/ROADMAP.md step marked complete
  • Unit tests for WarmupOptions
  • Integration test for WarmupRunner with SmolLM-135M

References

  • Roadmap: Step 46 (Phase 6 — Improved Serving)
  • Depends on: Step 34 (ASP.NET server) ✅
  • Design spec: docs/SERVER.md (Warm-up section)

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions