Skip to content

fix: update caliper postprocess config fields after PR 110 schema change - #118

Merged
openshift-merge-bot[bot] merged 3 commits into
openshift-psap:mainfrom
Harshith-umesh:fix/caliper-postprocess-config-fields
Jul 16, 2026
Merged

openshift-merge-bot[bot] merged 3 commits into
openshift-psap:mainfrom
Harshith-umesh:fix/caliper-postprocess-config-fields

Conversation

@Harshith-umesh

@Harshith-umesh Harshith-umesh commented Jul 9, 2026 •

Copy link
Copy Markdown
Member

PR 110 renamed caliper.postprocess.kpi fields:

  • kpi.generate -> kpi.artifacts_to_kpis
  • kpi.export -> removed (no longer needed)
  • analyze.baseline -> analyze.historical_kpis

These three project configs were not updated, causing pydantic ValidationError (extra_forbidden) during postprocessing.

Summary by CodeRabbit

  • New Features

    • Updated KPI processing to derive KPIs from available artifacts and write results to kpis.json.
    • Applied the same KPI output behavior across multiple project configurations for consistent reporting.
  • Bug Fixes

    • Simplified KPI analysis configuration by removing the old baseline-based setting and switching to historical_kpis: null while keeping analysis disabled.
  • Documentation

    • Expanded the GitOps documentation with clearer Tekton pipeline options and detailed FournosJob environment variable behavior for git checkout.

PR 110 renamed caliper.postprocess.kpi fields:
- kpi.generate -> kpi.artifacts_to_kpis
- kpi.export -> removed (no longer needed)
- analyze.baseline -> analyze.historical_kpis

These three project configs were not updated, causing
pydantic ValidationError (extra_forbidden) during postprocessing.

Co-authored-by: Cursor <cursoragent@cursor.com>
@coderabbitai

coderabbitai Bot commented Jul 9, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Updated Caliper KPI configuration in three project YAML files and expanded Fournos documentation with pipeline variants, Git checkout behavior, and configuration examples.

Changes

Configuration and documentation updates

Layer / File(s) Summary
Caliper KPI configuration migration
projects/llm_d_legacy/testing/config.yaml, projects/mcp_gateway/orchestration/config.yaml, projects/rhaiis/orchestration/config.yaml
Replaced the previous generate/export/baseline structure with artifacts_to_kpis, kpis.json output, and historical_kpis where applicable.
Fournos pipeline and Git checkout documentation
fournos/gitops/README.md
Documented pipeline variants and selection defaults, Git checkout environment variables, pull request result posting, and YAML configuration examples.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly describes the main config schema update caused by PR 110 and matches the core changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Harshith-umesh

Copy link
Copy Markdown
Member Author

Add pipeline reference table and spec.env documentation
(PULL_PULL_SHA, PULL_NUMBER, PULL_BASE_SHA) to the gitops README.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Harshith-umesh
Harshith-umesh force-pushed the fix/caliper-postprocess-config-fields branch from 4c36f76 to 18eab4e Compare July 9, 2026 20:35
@kpouget

kpouget commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

can you move the doc to docs/fournos/something?

@psap-forge-bot

psap-forge-bot Bot commented Jul 9, 2026

Copy link
Copy Markdown

🟢 Execution of rhaiis-hera-benchmark 🟢

Execution Engine Configuration

forge:
  args:
  - nvidia
  - ci-quick
  configOverrides:
    rhaiis.deploy.image_pull_secret: npalaska-image-pull
    rhaiis.deploy.storage_pvc: model-pvc
    rhaiis.namespace: kserve-e2e-perf
    tests.rhaiis.model_key: qwen3-0_6b
  project: rhaiis

Artifact Links

Test Logs

✅ 00 Pre-Cleanup 1 second

✅ 01 Prepare 2 seconds

✅ 02 Preflight 1 second

✅ 03 Test 4 minutes, 53 seconds

✅ 04 Post-Cleanup 3 seconds

🔄 05 Export-Artifacts

Post-processing Status ✅

  • ✅ parse: success
  • ✅ artifacts_to_kpis: success
  • ⏭️ kpis_to_csv: disabled

    kpi.kpis_to_csv disabled

  • ⏭️ artifacts_to_ai_data: disabled

    kpi.artifacts_to_ai_data disabled

  • ⏭️ s3_import: disabled

    s3_import disabled

  • ⏭️ analyse_kpis: disabled

    analyze disabled

  • ⏭️ s3_export: disabled

    s3_export disabled

@kpouget

kpouget commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 15, 2026
@kpouget

kpouget commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

/approve
/hold

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 15, 2026
@openshift-ci

openshift-ci Bot commented Jul 15, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: kpouget

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 15, 2026
@openshift-ci openshift-ci Bot removed the lgtm Indicates that a PR is ready to be merged. label Jul 16, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@fournos/gitops/README.md`:
- Around line 40-53: Update the documented default in the Workflows section to
`forge-full`, matching the existing pipeline name in the table and the
`spec.pipeline` configuration.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 515d5cb9-7c71-4b78-b32f-59f1588f59c4

📥 Commits

Reviewing files that changed from the base of the PR and between c4854cf and 381da53.

📒 Files selected for processing (1)
  • fournos/gitops/README.md

Comment thread fournos/gitops/README.md
Comment on lines +40 to +53
### Workflows
- **Task**: `forge-step` — reusable Tekton task for executing forge commands
- **Pipelines**:

| Pipeline | Steps | Finally | Use case |
|---|---|---|---|
| `forge-full` | pre-cleanup → prepare → preflight → test | post-cleanup → export-artifacts | Full lifecycle with cleanup on both ends |
| `forge-prepare-test` | prepare → preflight → test | export-artifacts | Prepare + test without cleanup |
| `forge-test-only` | preflight → test | export-artifacts | Skip preparation, just run the test |
| `forge-prepare-only` | prepare → preflight | export-artifacts | Set up the cluster without benchmarks |
| `forge-replot` | replot | export-artifacts | Re-run visualization/postprocessing |

Select a pipeline via `spec.pipeline` in the FournosJob CR (default: `fournos-full`).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== README excerpt =="
nl -ba fournos/gitops/README.md | sed -n '30,70p'

echo
echo "== Search for pipeline names =="
rg -n --hidden --no-ignore-vcs '\b(forge-full|fournos-full|forge-prepare-test|forge-test-only|forge-prepare-only|forge-replot)\b' fournos/gitops . || true

echo
echo "== List gitops files =="
git ls-files fournos/gitops

Repository: openshift-psap/forge

Length of output: 219


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== README excerpt =="
sed -n '30,70p' fournos/gitops/README.md | cat -n

echo
echo "== Search for pipeline names =="
if command -v rg >/dev/null 2>&1; then
  rg -n --hidden --no-ignore-vcs '\b(forge-full|fournos-full|forge-prepare-test|forge-test-only|forge-prepare-only|forge-replot)\b' fournos/gitops . || true
else
  grep -RInE '\b(forge-full|fournos-full|forge-prepare-test|forge-test-only|forge-prepare-only|forge-replot)\b' fournos/gitops . || true
fi

echo
echo "== git ls-files under fournos/gitops =="
git ls-files fournos/gitops

Repository: openshift-psap/forge

Length of output: 5923


Make the documented default forge-full.

The table lists forge-full, but the default line says fournos-full, which doesn’t match any workflow name and will mislead users.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@fournos/gitops/README.md` around lines 40 - 53, Update the documented default
in the Workflows section to `forge-full`, matching the existing pipeline name in
the table and the `spec.pipeline` configuration.

@kpouget

kpouget commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 16, 2026
@Harshith-umesh

Copy link
Copy Markdown
Member Author

/lgtm

@openshift-ci

openshift-ci Bot commented Jul 16, 2026

Copy link
Copy Markdown

@Harshith-umesh: you cannot LGTM your own PR.

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@Harshith-umesh

Copy link
Copy Markdown
Member Author

/hold cancel

@openshift-ci openshift-ci Bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 16, 2026
@openshift-merge-bot
openshift-merge-bot Bot merged commit 4d720e0 into openshift-psap:main Jul 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants