Skip to content

[llm_d] Work on P/D CPT - #126

Merged
openshift-merge-bot[bot] merged 17 commits into
openshift-psap:mainfrom
kpouget:llmd
Jul 23, 2026
Merged

openshift-merge-bot[bot] merged 17 commits into
openshift-psap:mainfrom
kpouget:llmd

Conversation

@kpouget

@kpouget kpouget commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features
    • Added configurable deployment templates for simple, intelligent-routing, and prefill/decode workloads.
    • Added prefill/decode deployment profiles with dedicated scheduling and scaling options.
    • Added Guidellm multiturn and heterogeneous evaluation profiles.
    • Enabled Kueue scheduling and configurable endpoint-picker routing.
  • Bug Fixes
    • Improved preset inheritance and deployment profile resolution.
  • Documentation
    • Added guidance for configuring and testing prefill/decode deployments.
  • Tests
    • Added coverage for deployment generation, routing configuration, and prefill/decode integration.

@coderabbitai

coderabbitai Bot commented Jul 15, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Template-based deployment profiles, P/D rendering, scheduler and Kueue configuration, Guidellm workloads, preset inheritance updates, generated-manifest references, and integration tests are added or updated for llm-d orchestration.

Changes

llm-d deployment orchestration

Layer / File(s) Summary
Profile templates and execution presets
projects/llm_d/orchestration/config.d/*, projects/llm_d/orchestration/config.yaml, projects/llm_d/orchestration/manifests/scheduler_config/*, projects/llm_d/orchestration/presets.d/presets.yaml, projects/core/library/config.py
Adds P/D, intelligent-routing, and simple profile templates; configures scheduler/EPP manifests, Kueue, Guidellm workloads, deployment profiles, smoke presets, and resilient preset inheritance.
Runtime resolution and manifest rendering
projects/llm_d/orchestration/runtime_config.py, projects/llm_d/orchestration/render_inference_service.py, projects/llm_d/orchestration/test_phase.py, projects/llm_d/test_epp_config.py
Resolves profile-name placeholders, loads scheduler templates, exposes P/D helpers, renders standard or P/D services, applies Kueue metadata, and propagates deployment profile names.
Generated deployment validation
projects/llm_d/tests/test_deployment_profiles.py, projects/llm_d/tests/test_pd_integration.py, projects/llm_d/tests/reference_deployments/*, projects/llm_d/docs/PD_TESTING.md
Adds preset-level generated-manifest comparisons, P/D configuration and matrix tests, reference manifests, and P/D testing documentation.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Preset
  participant RuntimeConfig
  participant Renderer
  participant Scheduler
  participant KServeManifest
  Preset->>RuntimeConfig: resolve deployment profile
  RuntimeConfig->>Scheduler: inject EPP configuration
  RuntimeConfig->>Renderer: provide resolved profile
  Renderer->>KServeManifest: render standard or P/D service
  Renderer->>KServeManifest: apply Kueue metadata
Loading

Possibly related PRs

Suggested labels: lgtm, approved

Suggested reviewers: albertoperdomo2

🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (1 warning, 1 inconclusive)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 70.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
Title check ❓ Inconclusive The title is related to P/D changes, but it is too vague to clearly summarize the main update. Use a more specific title like "[llm_d] Add P/D deployment and runtime support".
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kpouget

kpouget commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster forge-smoke-testing
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@kpouget

kpouget commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@psap-forge-bot

Copy link
Copy Markdown

🛑 Execution of llm_d smoke 🛑
🛑 JOB ABORTED - spec.shutdown=Stop

Execution Engine Configuration

forge:
  args:
  - smoke
  configOverrides:
    platform.operators.rhods-operator.channel: beta
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310
  project: llm_d

Artifact Links

Test Logs

✅ 001 Prepare 2 minutes, 39 seconds

❓ 002 Test

Test Description

This test validates the llm_d project by executing a smoke benchmark on the Qwen/Qwen3-0.6B model configured with the approximate-prefix-cache deployment profile, utilizing a short workload with small token sizes and low concurrency rates to verify basic inference functionality.

🔄 003 Export-Artifacts

@kpouget

kpouget commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@psap-forge-bot

Copy link
Copy Markdown

🔴 Execution of llm_d smoke 🔴

Execution Engine Configuration

forge:
  args:
  - smoke
  configOverrides:
    platform.operators.rhods-operator.channel: beta
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310
  project: llm_d

Artifact Links

Test Logs

❌ 001 Prepare 21 minutes, 51 seconds

Failure Review 013 Cluster Deploy Operator Rhods-Operator

013__cluster_deploy_operator__rhods-operator

The wait_for_csv_ready task failed because the rhods-operator.3.5.0-ea.2 ClusterServiceVersion (CSV) entered a Failed state after repeatedly cycling between Installing and InstallReady for approximately 11 minutes. The root cause is an OLM Operator Deployment Reconciliation Failure where the operator Deployment could not reach a ready state despite the InstallPlan being successfully applied and approved, causing OLM to abort the installation cycle.

🔄 003 Export-Artifacts

@psap-forge-bot

Copy link
Copy Markdown
🔴 Submission of llm_d smoke failed after 23 minutes, 29 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20260715-122856' failed: Tasks Completed: 2 (Failed: 1, Cancelled 0), Skipped: 1

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@kpouget

kpouget commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@psap-forge-bot

Copy link
Copy Markdown

🔴 Execution of llm_d smoke 🔴

Execution Engine Configuration

forge:
  args:
  - smoke
  configOverrides:
    platform.operators.rhods-operator.channel: beta
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310
  project: llm_d

Artifact Links

Test Logs

❌ 001 Prepare 16 minutes, 3 seconds

Failure Review 013 Cluster Deploy Operator Rhods-Operator

013__cluster_deploy_operator__rhods-operator

The wait_for_csv_ready task failed, aborting the deployment of rhods-operator.3.5.0-ea.2, because the ClusterServiceVersion transitioned to a terminal Failed state after OLM could not complete resource reconciliation within the timeout window. Analysis confirms the CSV remained stuck in the Installing phase for approximately 10.5 minutes with intermittent retry cycles, causing OLM to explicitly mark the installation as failed.

🔄 003 Export-Artifacts

@psap-forge-bot

Copy link
Copy Markdown
🔴 Submission of llm_d smoke failed after 19 minutes, 10 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20260715-130528' failed: Tasks Completed: 2 (Failed: 1, Cancelled 0), Skipped: 1

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@kpouget

kpouget commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@psap-forge-bot

Copy link
Copy Markdown

🔴 Execution of llm_d smoke 🔴

Execution Engine Configuration

forge:
  args:
  - smoke
  configOverrides:
    platform.operators.rhods-operator.channel: beta
    platform.rhoai.custom_catalog.enabled: true
    platform.rhoai.custom_catalog.image: quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310
  project: llm_d

Artifact Links

Test Logs

✅ 001 Prepare 22 minutes, 47 seconds

❌ 002 Test 16 minutes, 34 seconds

Test Description

This test evaluates the llm_d project by deploying the Qwen/Qwen3-0.6B model using the approximate prefix caching profile and executing a quick smoke test alongside a short concurrent benchmark (256 prompt/128 output tokens at rates 1–8). It specifically validates inference functionality and performance under the approximate-prefix-cache deployment configuration.

Failure Review 003 Run Smoke Request

001__llmd_test/003__run_smoke_request

The wait_smoke_pod_running task failed after 60 retries because the llm-d-smoke pod remained stuck in the Pending phase, completely blocking the smoke test workflow. The definitive root cause is a Kubernetes scheduler blockage where the pod was admitted to the API server but could not be assigned to any available node; analysis of src/smoke-pod.yaml confirms the manifest lacks explicit resource requests or node selectors, indicating the failure is caused by namespace-level constraints such as ResourceQuota exhaustion, LimitRange defaults, or a mismatch between the pod's scheduling requirements and available cluster node labels or taints.

🔄 003 Export-Artifacts

Post-processing Status ❌

@psap-forge-bot

Copy link
Copy Markdown
🔴 Submission of llm_d smoke failed after 42 minutes, 20 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20260715-141441' failed: Tasks Completed: 3 (Failed: 1, Cancelled 0), Skipped: 0

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-prepare-test
/rhoai.rc-image quay.io/rhoai/rhoai-fbc-fragment@sha256:e213152dc5bdaaf5724d269484914f3e1d7b4a4959d4405cb3eca9dd6297b310

@kpouget
kpouget force-pushed the llmd branch 2 times, most recently from a94115b to e503280 Compare July 15, 2026 15:41
@kpouget

kpouget commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-test-only

@psap-forge-bot

Copy link
Copy Markdown

🟢 Execution of llm_d smoke 🟢

Execution Engine Configuration

forge:
  args:
  - smoke
  configOverrides: {}
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 2 seconds

✅ 01 Test 11 minutes, 10 seconds

Test Description

This test performs a smoke validation for the llm_d project by deploying the Qwen/Qwen3-0.6B model using the approximate-prefix-cache deployment profile and running a short concurrent benchmark (1–8 requests/sec) to verify basic inference functionality and gateway connectivity.

🔄 02 Export-Artifacts

Post-processing Status ✅

@psap-forge-bot

Copy link
Copy Markdown
🟢 Submission of llm_d smoke succeeded after 12 minutes, 56 seconds 🟢
/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-test-only

@kpouget

kpouget commented Jul 16, 2026 •

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-test-only
/var runtime.kserve_dry_run: true
/fournos wip

@kpouget

kpouget commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-test-only
/var runtime.kserve_dry_run: true
/fournos wip

@psap-forge-bot

Copy link
Copy Markdown
🔴 Submission of llm_d smoke failed after 2 minutes, 17 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20260716-124030' failed: Job not found - may have been deleted or never created

/test fournos llm_d smoke
/var runtime.kserve_dry_run: true
/cluster kpouget-llmd
/pipeline forge-test-only

@kpouget

kpouget commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-test-only
/var runtime.kserve_dry_run: true
/var fournos.namespace: psap-automation-wip

@psap-forge-bot

Copy link
Copy Markdown
🔴 Submission of llm_d smoke failed after 5 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20260716-124342' failed: Failed to fetch Pipeline 'forge-test-only': Not Found

/test fournos llm_d smoke
/var runtime.kserve_dry_run: true
/var fournos.namespace: psap-automation-wip
/cluster kpouget-llmd
/pipeline forge-test-only

@kpouget
kpouget force-pushed the llmd branch 2 times, most recently from 76f89ea to 94d503b Compare July 16, 2026 13:43
@kpouget

kpouget commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d smoke
/cluster kpouget-llmd
/pipeline forge-test-only
/var runtime.kserve_dry_run: true
/fournos wip

@psap-forge-bot

Copy link
Copy Markdown

🟢 Execution of llm_d smoke 🟢

Execution Engine Configuration

forge:
  args:
  - smoke
  configOverrides:
    runtime.kserve_dry_run: true
  project: llm_d

Artifact Links

Test Logs

✅ 00 Preflight 5 seconds

✅ 01 Test 4 seconds

🔄 02 Export-Artifacts

@psap-forge-bot

Copy link
Copy Markdown
🟢 Submission of llm_d smoke succeeded after 3 minutes, 30 seconds 🟢
/test fournos llm_d smoke
/var runtime.kserve_dry_run: true
/cluster kpouget-llmd
/pipeline forge-test-only
/fournos wip

@kpouget
kpouget force-pushed the llmd branch 3 times, most recently from 64caa2d to 9245d42 Compare July 22, 2026 10:00
vllm_default_args = _build_vllm_args(
config.project.get_config("deployments.defaults.vllm_extra.args")
)
vllm_deploy_args = _build_vllm_args(deployment_profile.get("vllm_extra.args", {}))

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this dict.get() work? I think the deployment profile is a nested dict: {"vllm_extra": {"args": [...]}} so I'm not sure if this is working.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, it "works" (doesn't crash), but doesn't do anything useful 😅

actually, it went unnoticed because the default args are already applied earlier in the runtime config, so this call is redundant

fixed by removing deployments.defaults.vllm_extra.args and properly fetching vllm_extra.args

Comment thread projects/llm_d/orchestration/config.d/platform.yaml Outdated
@albertoperdomo2

Copy link
Copy Markdown
Collaborator

/lgtm
/approve

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 23, 2026
@openshift-ci

openshift-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: albertoperdomo2

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 23, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit 6dfbb35 into openshift-psap:main Jul 23, 2026
6 checks passed
@kpouget
kpouget deleted the llmd branch July 30, 2026 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants