Repository navigation
Port All model furnace features to unified tooling - #128
openshift-merge-bot[bot] merged 62 commits into
Conversation
|
Skipping CI for Draft Pull Request. |
openshift-ci
Bot
added
the
do-not-merge/work-in-progress
|
Warning Review limit reached
Next review available in: 58 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (6)
📝 WalkthroughWalkthroughAdds multi-workload RHAIIS benchmarking with profiler workflows, curve-based KPI exports, S3 dashboard synchronization, regression and AI-agent analysis, pipeline notifications, and improved GuideLLM credential and result handling. ChangesRHAIIS benchmarking and regression pipeline
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant test_phase
participant run_guidellm_benchmark
participant RhaiisParser
participant RhaiisPlugin
participant run_regression_check
participant run_agent_analysis
test_phase->>run_guidellm_benchmark: run workload benchmark
run_guidellm_benchmark-->>test_phase: return benchmark artifacts
test_phase->>RhaiisParser: parse benchmark metrics and curves
RhaiisParser->>RhaiisPlugin: emit KPI records
RhaiisPlugin->>run_regression_check: provide dashboard CSV
run_regression_check->>run_agent_analysis: analyze severe regressions
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
- CSV dashboard export (csv_export.py): generates consolidated_dashboard.csv from PSAP JSON files matching model-furnace's exact column schema - S3 sync (s3_dashboard.py): download/append/re-upload consolidated CSV and upload profiler traces, using new psap-forge-dashboard-s3 vault - Regression detection (analyze.py): geometric mean comparison with configurable thresholds per metric, ported from model-furnace's compare_results.py - Slack webhook notifications (send.py): regression/improvement alerts via Incoming Webhook (psap-forge-rhaiis-slack vault), with Bot Token fallback - Profiler toolbox tasks: enable/disable gate, copy traces from pod, and verify_profiler_prereqs (webhook pod, TLS, ConfigMap checks) - Test phase wiring: profiler prereq check → gate toggle → benchmark → trace copy → PSAP payload → CSV export → S3 sync → regression analysis → Slack → profiler trace upload → MLflow metadata - Config: new vault definitions, profiler section, csv_dashboard and analyze sections in rhaiis config Co-authored-by: Cursor <cursoragent@cursor.com>
- Fix CSV export to match model-furnace field extraction exactly:
accelerator parsing, runtime_args formatting, metric key paths,
raw ms values (no /1000), median/p01 instead of p50/p1, and
add missing columns (DP, dataset, spec_decoding, etc.)
- Generate a single UUID4 per run and thread it through PSAP payload
(experiment_id), dashboard CSV (uuid column), predictor log upload
(logs/{uuid}.log on S3), and regression notifications (job_id)
- Add predictor pod log upload to s3://psap-model-furnace/logs/
Co-authored-by: Cursor <cursoragent@cursor.com>
The openshift-client tarball includes both oc and kubectl, but the Containerfile was deleting kubectl after extraction. Pipeline tasks that reference kubectl (e.g. 00-pre-cleanup) fail with exit code 127. Co-authored-by: Cursor <cursoragent@cursor.com>
This reverts commit e17f171.
- Use gpu_type (e.g. H200) instead of platform name (nvidia) for PSAP payload, CSV, and S3 artifact paths - Exclude warmup benchmarks.json from PSAP payload generation to prevent warmup results contaminating the dashboard CSV Co-authored-by: Cursor <cursoragent@cursor.com>
Track the main benchmark artifact directory explicitly by index instead of globbing for all benchmarks.json files. Warmup and profiler benchmark results are no longer included in the PSAP payload or dashboard CSV. Co-authored-by: Cursor <cursoragent@cursor.com>
When rhaiis.cluster_tag is set (e.g. "hera2"), the PSAP payload accelerator_type becomes "H200_HERA2", which csv_export splits into accelerator=H200 and run=H200-hera2-model-TP. Co-authored-by: Cursor <cursoragent@cursor.com>
- merge_vllm_args now applies rhaiis.vllm_args last so configOverrides like tensor-parallel-size take precedence over model defaults - Remove tensor-parallel-size from rhaiis.yaml defaults (models define it) - Move namespace and image_pull_secret into rhaiis.yaml defaults - Fix predictor log upload bucket to psap-dashboard-data Co-authored-by: Cursor <cursoragent@cursor.com>
The capture_isvc_state toolbox task writes logs under its own sub-directory (NNN__capture_isvc_state/artifacts/), not the top-level ARTIFACT_DIR. Co-authored-by: Cursor <cursoragent@cursor.com>
- Add rhaiis.warmup config section (rate: 200, max_seconds: 60) - Add profiler rates/max_seconds to config (200/200, matching model-furnace) - Read warmup params from config instead of hardcoding Co-authored-by: Cursor <cursoragent@cursor.com>
Derives labels like isl1000_osl1000 from the workload's data field instead of requiring manual rhaiis.profiler.labels config, matching model-furnace's get_profile_label() behavior. Co-authored-by: Cursor <cursoragent@cursor.com>
…plugin Move compare_runs, run_regression_analysis, RegressionResult, and geometric_mean out of the generic caliper/engine/kpi/analyze.py into rhaiis/postprocess/regression.py. The caliper engine should contain no project-specific code (no references to model, accelerator, TP, tokens, etc.). This is a temporary transition — these CSV-based comparison functions will be replaced by generic Caliper KPI-based analysis once the framework lands. Co-authored-by: Cursor <cursoragent@cursor.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (4)
projects/rhaiis/postprocess/csv_export.py (1)
137-146: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winPreserve all
report.args.dataentries when extracting token counts.
generate_dashboard_csv()applies the same token metadata to every CSV row, butreport.args.datacan contain repeated--dataentries; dropping all rows after the first can assign the wrongprompt_toks/output_toksto benchmarks that used a later data source. Aggregate or fail explicitly when multiple data configurations are present instead of silently using only the first string.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@projects/rhaiis/postprocess/csv_export.py` around lines 137 - 146, Update the token extraction in generate_dashboard_csv() so every report.args.data entry is preserved and considered, rather than selecting only data_list[0]. Aggregate token metadata across entries or explicitly reject multiple configurations; do not silently discard later entries or assign token counts from only the first data source.projects/rhaiis/orchestration/test_phase.py (2)
158-181: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy liftModule-global
_warningsstate still unresolved, including the standalone-analysis early return.
_warningsremains process-global mutable state accumulated across_run_test/_run_workload_benchmark. The standalone-analysis early return at Line 180 (return 1 if _warnings else 0) still doesn't clear_warningsbefore returning, exactly the gap called out in a prior review ("never cleared in the standalone-analysis early return"). If two test invocations share a process, warnings from a previous run leak into the next run's return code and Slack notification.Also applies to: 312-326
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@projects/rhaiis/orchestration/test_phase.py` around lines 158 - 181, Remove the process-global `_warnings` dependency from `_run_test` and `_run_workload_benchmark` by creating and using per-invocation warning state. Ensure the standalone-analysis early return and the corresponding flow around lines 312–326 evaluate only the current invocation’s warnings, so prior runs cannot affect return codes or Slack notifications.
709-734: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winName copied trace files by workload before merging.
Traces are copied from different
*__copy_profiler_tracesdirectories and merged into the sametraces_combineddirectory usingf.nameonly. Names liketrace_rank*.jsoncan collide across workloads, causing files to silently overwrite each other instead of preserving one trace per workload/profile. Prefix the target paths with the parent directory name or a workload/profile token, and apply the same namespace when inferred labels are used.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@projects/rhaiis/orchestration/test_phase.py` around lines 709 - 734, Update the merge logic in _upload_profiler_traces so copied trace filenames include a workload/profile namespace derived from each source parent directory, preventing collisions in traces_combined. Use the same prefixed naming when selecting or inferring labels from merged traces, while preserving direct uploads when all traces share one directory.projects/rhaiis/postprocess/regression.py (1)
380-418: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
status == "failed"is silently swallowed with zero logging.
send_regression_notificationexplicitly logs and returns for"skipped", but any other non-"completed"status — including"failed", whichrun_regression_analysis's exception handler produces — falls through the genericif status != "completed": return Truewith no log line at all. A regression-analysis crash is thus reported as a successful notification and leaves no trace, which contradicts the PR stack's repeated goal of not swallowing errors silently.🛠️ Proposed fix
if status == "skipped": reason = analysis_result.get("reason", "unknown") logger.info("Regression analysis skipped: %s", reason) return True - if status != "completed": + if status == "failed": + logger.warning( + "Regression analysis failed, no notification sent: %s", + analysis_result.get("error", "unknown"), + ) + return False + + if status != "completed": return True🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@projects/rhaiis/postprocess/regression.py` around lines 380 - 418, Update send_regression_notification so status == "failed" is logged with its available error or failure details before returning. Preserve the existing skipped handling, and ensure all non-completed statuses are not silently swallowed while retaining the current return behavior unless the surrounding contract requires otherwise.
♻️ Duplicate comments (1)
projects/rhaiis/orchestration/test_phase.py (1)
100-103: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy liftMulti-workload runs still deploy/profile using only
workload_keys[0]'svllm_args.
vllm_argsis derived once fromfirst_workload(Line 102-103) and reused for the single deployment and for every workload's profiler/warmup step in the Phase 1 loop (Lines 232-261). Any additional workload with differentvllm_argsis silently ignored, matching a previously raised major concern that is still unaddressed here.Also applies to: 232-261
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@projects/rhaiis/orchestration/test_phase.py` around lines 100 - 103, Update the Phase 1 deployment and profiling flow to derive merged vLLM arguments separately for each workload instead of reusing arguments built from workload_keys[0]. Use each workload’s configuration when invoking deployment, profiler, and warmup steps, while preserving shared defaults and image selection.
🧹 Nitpick comments (1)
projects/rhaiis/postprocess/regression.py (1)
517-595: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winConsider guarding against secret leakage before forwarding raw error text to Slack.
error_text(and equivalents insend_regression_notification) is forwarded verbatim (only length-truncated) into an outbound Slack message. If an upstream failure ever embeds credential-like content (e.g. a raw exception from an S3/vault call), it would be posted externally. As per coding guidelines, "Never print, log, or include secret values in error messages," consider a lightweight redaction pass (e.g. common token/key patterns) before composing the message.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@projects/rhaiis/postprocess/regression.py` around lines 517 - 595, Update send_failure_notification and the equivalent error-text handling in send_regression_notification to redact common credential and token patterns before truncation and Slack message composition. Preserve the existing length limit and notification flow, applying the redaction to all raw upstream error text forwarded externally.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@projects/rhaiis/postprocess/csv_export.py`:
- Around line 91-102: Update _format_runtime_args to prevent sensitive
inference_server_args and trtllm_config entries from being written to
runtime_args in consolidated_dashboard.csv. Mask or skip credential, API-key,
and token-bearing keys before appending formatted values, or apply an existing
dashboard-safe allowlist while preserving non-sensitive arguments.
In `@projects/rhaiis/postprocess/regression.py`:
- Around line 100-105: Update the object-column normalization loops in the
postprocessing function to strip whitespace only from values that are actual
strings, preserving numeric and other non-string values unchanged. Apply the
same guarded normalization to both current_df and baseline_df before the later
TP, version, and accelerator filtering.
---
Outside diff comments:
In `@projects/rhaiis/orchestration/test_phase.py`:
- Around line 158-181: Remove the process-global `_warnings` dependency from
`_run_test` and `_run_workload_benchmark` by creating and using per-invocation
warning state. Ensure the standalone-analysis early return and the corresponding
flow around lines 312–326 evaluate only the current invocation’s warnings, so
prior runs cannot affect return codes or Slack notifications.
- Around line 709-734: Update the merge logic in _upload_profiler_traces so
copied trace filenames include a workload/profile namespace derived from each
source parent directory, preventing collisions in traces_combined. Use the same
prefixed naming when selecting or inferring labels from merged traces, while
preserving direct uploads when all traces share one directory.
In `@projects/rhaiis/postprocess/csv_export.py`:
- Around line 137-146: Update the token extraction in generate_dashboard_csv()
so every report.args.data entry is preserved and considered, rather than
selecting only data_list[0]. Aggregate token metadata across entries or
explicitly reject multiple configurations; do not silently discard later entries
or assign token counts from only the first data source.
In `@projects/rhaiis/postprocess/regression.py`:
- Around line 380-418: Update send_regression_notification so status == "failed"
is logged with its available error or failure details before returning. Preserve
the existing skipped handling, and ensure all non-completed statuses are not
silently swallowed while retaining the current return behavior unless the
surrounding contract requires otherwise.
---
Duplicate comments:
In `@projects/rhaiis/orchestration/test_phase.py`:
- Around line 100-103: Update the Phase 1 deployment and profiling flow to
derive merged vLLM arguments separately for each workload instead of reusing
arguments built from workload_keys[0]. Use each workload’s configuration when
invoking deployment, profiler, and warmup steps, while preserving shared
defaults and image selection.
---
Nitpick comments:
In `@projects/rhaiis/postprocess/regression.py`:
- Around line 517-595: Update send_failure_notification and the equivalent
error-text handling in send_regression_notification to redact common credential
and token patterns before truncation and Slack message composition. Preserve the
existing length limit and notification flow, applying the redaction to all raw
upstream error text forwarded externally.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 757d23c1-6548-4301-b338-2e9765d29770
📒 Files selected for processing (14)
projects/caliper/engine/kpi/analyze.pyprojects/guidellm/toolbox/run_guidellm_benchmark/main.pyprojects/rhaiis/orchestration/analysis.pyprojects/rhaiis/orchestration/ci.pyprojects/rhaiis/orchestration/test_phase.pyprojects/rhaiis/postprocess/agent.pyprojects/rhaiis/postprocess/csv_export.pyprojects/rhaiis/postprocess/kpis.pyprojects/rhaiis/postprocess/parser.pyprojects/rhaiis/postprocess/plugin.pyprojects/rhaiis/postprocess/regression.pyprojects/rhaiis/postprocess/s3_dashboard.pyprojects/rhaiis/toolbox/enable_profiler_gate/main.pyprojects/rhaiis/toolbox/verify_profiler_prereqs/main.py
🚧 Files skipped from review as they are similar to previous changes (10)
- projects/rhaiis/orchestration/ci.py
- projects/rhaiis/toolbox/enable_profiler_gate/main.py
- projects/rhaiis/toolbox/verify_profiler_prereqs/main.py
- projects/rhaiis/postprocess/plugin.py
- projects/rhaiis/postprocess/kpis.py
- projects/rhaiis/postprocess/parser.py
- projects/guidellm/toolbox/run_guidellm_benchmark/main.py
- projects/rhaiis/postprocess/s3_dashboard.py
- projects/rhaiis/postprocess/agent.py
- projects/rhaiis/orchestration/analysis.py
…rofile data
Previously, all workload benchmarks ran under a single NextArtifactDir
("testing") with one set of test labels (workload_key=profile1). The
Caliper parser merged all benchmarks.json files into one record, causing:
- Dashboard showing only "profile1" regardless of which profiles ran
- Multiple data points on the same chart from different profiles
Now each workload's benchmark phase runs in its own NextArtifactDir
(e.g., "benchmark_profile1", "benchmark_profile2") with per-workload
test labels. The parser creates separate records per profile, producing
correct per-profile CSV rows and dashboard entries.
Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
projects/rhaiis/orchestration/test_phase.py (1)
219-244: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy liftMulti-workload phase loops have no per-workload failure isolation.
Neither the phase-1 (warmup/profiler) loop nor the phase-2 (
_run_workload_benchmark) loop wraps each workload iteration in its own try/except. If workload N raises (e.g. guidellm benchmark error, profiler step failure), the exception propagates straight to the outerfinally: _capture_and_cleanup(...)and then todo_test()'s pipeline-failure handling, aborting all subsequent workloads inworkload_keys. For a multi-workload benchmark run, this means one workload's failure discards results for every workload that hasn't run yet, rather than collecting per-workload results/warnings and continuing.Consider catching per-workload exceptions inside each loop iteration, recording them into the warnings list, and continuing to the next
wl_keyso partial results are preserved and reported.Also applies to: 253-271
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@projects/rhaiis/orchestration/test_phase.py` around lines 219 - 244, Update both workload loops around the phase-1 profiler/warmup calls and phase-2 _run_workload_benchmark invocation to catch exceptions per wl_key iteration, append each failure to the existing warnings collection, and continue processing subsequent workloads. Keep successful workload result handling unchanged and ensure the captured warnings remain included in the final partial-results reporting and cleanup flow.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@projects/rhaiis/orchestration/test_phase.py`:
- Around line 370-378: Update the run_standalone_analysis call in the
not-run_benchmark branch to pass restrict_profiles scoped to the current
workload_key. Preserve the existing model_cfg, accelerator_key, vllm_args, and
run_uuid arguments while ensuring standalone analysis processes only that
workload.
---
Outside diff comments:
In `@projects/rhaiis/orchestration/test_phase.py`:
- Around line 219-244: Update both workload loops around the phase-1
profiler/warmup calls and phase-2 _run_workload_benchmark invocation to catch
exceptions per wl_key iteration, append each failure to the existing warnings
collection, and continue processing subsequent workloads. Keep successful
workload result handling unchanged and ensure the captured warnings remain
included in the final partial-results reporting and cleanup flow.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 9c609093-5b03-4ad1-8716-b01392c90326
📒 Files selected for processing (1)
projects/rhaiis/orchestration/test_phase.py
| @ci_lib.safe_ci_entrypoint | ||
| def post_cleanup(ctx): | ||
| """Post-cleanup phase - Clean up resources after test.""" | ||
| _check_pipeline_failure_and_notify() |
There was a problem hiding this comment.
this belongs to the test step, no? 🤔
There was a problem hiding this comment.
this is to catch errors before the test step in the tekton pipeline, eg: image pull timeout or prepare step crashes. I would want to get slack notifications
There was a problem hiding this comment.
this is the post-cleanup 🤔
to catch things before the test step, you need to implement the preflight check, that's exactly the point of this function
| mgmt_env = {k: v for k, v in os.environ.items() if k != "KUBECONFIG"} | ||
| subprocess.run( | ||
| [ | ||
| "oc", | ||
| "annotate", | ||
| "fournosjob", | ||
| fjob_name, | ||
| "-n", | ||
| fjob_ns, | ||
| f"rhaiis.run-uuid={run_uuid}", | ||
| "--overwrite", | ||
| ], | ||
| check=False, | ||
| capture_output=True, | ||
| timeout=10, | ||
| env=mgmt_env, | ||
| ) |
There was a problem hiding this comment.
suggestion: use the run.run library file for clean log tracking
There was a problem hiding this comment.
good suggestion, but run.run doesn't support env overrides or timeout.
- Use pd.to_numeric for TP filtering in analysis.py to handle non-numeric values - Guard against empty ARTIFACT_BASE_DIR in ci.py pipeline failure detection - Anchor resource cleanup suffix match with endswith in prepare_rhaiis.py - Clear _warnings at start of _run_test to prevent cross-invocation leaks - Add pipefail and post-extraction verification to profiler trace copy - Pass restrict_profiles=[workload_key] in per-workload standalone analysis - Deduplicate notification functions into shared _send_alert helper - Stop swallowing exceptions: surface CSV sync, profiler upload failures - Use descriptive artifact dir name with model_key and workload_keys - Factor shared warmup/profiler step args into step_kwargs dict Co-authored-by: Cursor <cursoragent@cursor.com>
|
/test fournos rhaiis nvidia |
|
/test fournos rhaiis nvidia |
🟢 Execution of
|
🟢 Submission of
|
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kpouget The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
2 participants
Port model-furnace benchmarking features to forge/rhaiis
This PR ports the full RHAIIS benchmarking, analysis, and notification pipeline from the legacy
model-furnacerepository into the unifiedforgetooling, and integrates it with Caliper's postprocessing framework.What's included
Benchmarking & execution
benchmarks.guidellm.hf_token_secretconfig, conditionally injected into K8s job manifestslabelsdict parameter instead of hardcoded valuesoc cpwith gzip fallback forbenchmarks.jsonfiles >70MB that fail withunexpected EOFDashboard CSV generation (Caliper integration)
benchmarks.jsonlookup and PSAP payload generation with Caliper's plugin postprocessing pipeline: parse → KPI → CSV exportRhaiisParserextendsGuideLLMParserwith all per-benchmark extra metrics (p1/p999/mean latencies, token counts, throughput, concurrency, request counts)RhaiisKpiHandleremits per-rate-point KPIs covering all 31 dashboard CSV columnsRhaiisPlugin.export_kpis_to_csvpivots KPI records into the dashboard CSV schema with correct units (seconds→ms conversion for latency columns)request_rateto match GuideLLM parser ordering, preventing index misalignment_sync_postprocessed_dashboard_csvsyncs the generated CSV to S3 and triggers regression checks after Caliper postprocessingRegression analysis
compare_runsandrun_regression_analysisacceptprofile_mapandmetricsas parameters (no RHAIIS-specific code in caliper engine)Notifications
psap-rhaiis-alertschannelAI agent integration
PyTorch profiling pipeline
verify_profiler_prereqs: checks webhook pod status invllm-profilernamespace, validates MutatingWebhookConfiguration, TLS secret, and ConfigMap before profiler runsenable_profiler_gate: toggles the vLLM profiler webhook gate annotation on the InferenceService to start/stop trace collection per labelcopy_profiler_traces: extracts PyTorch profiler trace files from the predictor podpsap-dashboard-databucket,pytorch-profiles/rhaiisprefix)rhaiis.profilerconfig sectionvllm-profiler/enabled: "true"label (now configurable via the generic labels dict)S3 and config
rhaiis.s3config sectioncaliper.postprocess.csv_dashboardconfigpsap-forge-dashboard-s3(generic), consolidatedpsap-forge-notificationsCleanup and parallel safety
Review comments addressed
caliper/engineandcore/notificationsintorhaiis/postprocessorchestration/analysis.pyand failure alerts toorchestration/notifications.pyenable_profiler_gatescript argumentsDEFAULT_AGENT_URLmode="w+b"fromNamedTemporaryFilecallslabelsdictFiles changed (30 files, +3095 / -400)
test_phase.py,analysis.py,notifications.py,ci.py,prepare_rhaiis.pyparser.py,kpis.py,plugin.py,csv_export.py,s3_dashboard.py,regression.py,agent.pyrun_guidellm_benchmark/main.py,utils.py,guidellm_job.yaml.j2deploy_kserve_isvc/main.py,inferenceservice.yaml.j2,servingruntime.yaml.j2enable_profiler_gate/main.py,copy_profiler_traces/main.py,verify_profiler_prereqs/main.pyconfig.yaml,config.d/rhaiis.yaml,config.d/workloads.yamlkpi/analyze.pypsap-forge-dashboard-s3.yamlSummary by CodeRabbit