Skip to content

[fournos launcher] Improve the toolbox - #17

Merged
kpouget merged 16 commits into
openshift-psap:mainfrom
kpouget:launcher
Apr 8, 2026
Merged

kpouget merged 16 commits into
openshift-psap:mainfrom
kpouget:launcher

Conversation

@kpouget

@kpouget kpouget commented Apr 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • New Features

    • New toolbox CLI utilities and a FOURNOS "submit-and-wait" command to generate job manifests, submit jobs, poll for completion, and capture logs.
  • Bug Fixes

    • CI scripts must be executable before being run.
    • Shell command failures now show clearer messages including exit code and trimmed stdout/stderr.
  • Improvements

    • CI metadata and config overrides are stored in a standardized metadata directory.
    • Error summaries and task-execution reporting are more detailed and user-friendly.
    • Launcher defaults updated (cluster, job, project, namespace).

@openshift-ci

openshift-ci Bot commented Apr 7, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign sjmonson for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Apr 7, 2026 •

Copy link
Copy Markdown

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

Renames CI metadata folder to 000__ci_metadata, centralizes metadata file paths, adds a toolbox CLI framework, introduces TaskExecutionError and richer runtime logging, enforces executable CI script discovery, and implements a task-driven FOURNOS submit-and-wait flow with new entrypoints.

Changes

Cohort / File(s) Summary
CI metadata & config
projects/core/ci_entrypoint/prepare_ci.py, projects/core/library/config.py, projects/core/notifications/send.py
Replaced _meta references with 000__ci_metadata constant and updated read/write locations for PR artifacts, pr_config.txt, and variable_overrides.yaml.
CI script discovery
projects/core/ci_entrypoint/run_ci.py
find_ci_script() now requires candidate paths to be executable (os.X_OK) as well as existent.
DSL surface & logging banner
projects/core/dsl/__init__.py, projects/core/dsl/log.py
Imported new toolbox submodule; log_execution_banner gains optional log_file and prints the ARGUMENTS section only when args are present.
Runtime/task error handling & logging
projects/core/dsl/runtime.py, projects/core/dsl/task.py, projects/core/dsl/shell.py
Added TaskExecutionError, improved condition/task failure messages, moved task log to ARTIFACT_DIR/task.log, changed subprocess error message composition, refactored retry execution, and ensured file-handler cleanup.
Toolbox CLI framework
projects/core/dsl/toolbox.py
New module implementing positional-arg detection, dynamic CLI parsing, run_toolbox_command, get_task_execution_error, and create_toolbox_main for consistent CLI entrypoints and error reporting.
CI failure reporting
projects/core/library/ci.py
Reworked error handling to _display_error_summary and centralized _write_error_summary_to_file, producing structured, line-by-line FAILURE reports including task-specific sections.
Temp artifact teardown
projects/core/library/env.py
Removed exception-logging and automatic FAILURE-file writes from TempArtifactDir.__exit__; normal teardown still restores ARTIFACT_DIR.
FOURNOS orchestration API & config
projects/fournos_launcher/orchestration/submit.py, projects/fournos_launcher/orchestration/ci.py, projects/fournos_launcher/orchestration/cli.py, projects/fournos_launcher/orchestration/config.yaml
Renamed submit() → submit_job() and updated callers; updated orchestration defaults and added fournos.namespace in config.
New FOURNOS submit-and-wait flow
projects/fournos_launcher/toolbox/submit_and_wait/main.py
Added task-based run(...) that builds/applies manifests, polls job phase with retries, captures logs/YAML artifacts, cleans up manifest, and exposes main via toolbox.

Sequence Diagram

sequenceDiagram
    participant User
    participant CLI
    participant Toolbox
    participant DSL
    participant OC
    participant JobAPI

    User->>CLI: invoke submit_job()
    CLI->>Toolbox: toolbox entrypoint (create_toolbox_main)
    Toolbox->>Toolbox: parse args, env.init()
    Toolbox->>DSL: execute_tasks(run, locals())
    DSL->>DSL: validate inputs, compute job_name
    DSL->>OC: check oc availability (oc version)
    OC-->>DSL: success/fail
    DSL->>DSL: generate job manifest, write artifact
    DSL->>OC: oc apply -f manifest.yaml
    OC->>JobAPI: create job
    JobAPI-->>OC: job created
    OC-->>DSL: apply result
    loop poll until terminal
      DSL->>OC: oc get fournosjob ... -o jsonpath=status.phase
      OC->>JobAPI: query status
      JobAPI-->>OC: phase (Running/Completed/Failed)
      OC-->>DSL: phase
      DSL->>DSL: evaluate phase (retry/raise/succeed)
    end
    DSL->>OC: oc logs (retrieve)
    OC-->>DSL: logs
    DSL->>DSL: write logs and final YAML artifacts
    DSL->>DSL: cleanup manifest file (always)
    DSL-->>Toolbox: tasks complete
    Toolbox->>User: print success message / exit 0
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

Poem

🐰
I hopped the _meta path to 000__ci_metadata bright,
Tasks now whisper truths in logs deep through the night.
Toolbox trims the args, DSL tends the chore,
FOURNOS jobs take off and land with artifacts galore.
A rabbit bows—CI tidy, cozy, and right.

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 77.36% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title '[fournos launcher] Improve the toolbox' directly corresponds to the PR's primary objectives and the substantial changes to fournos launcher components and toolbox improvements demonstrated in the file changes.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 02 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator: Empty. (See run.log)

@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 03 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator: Empty. (See run.log)

@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 02 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator:

/logs/artifacts/FAILURE | ================================================================================
🚨 CI EXECUTION FAILED
================================================================================

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
~~ projects/fournos_launcher/toolbox/submit_and_wait/main.py:133
~~ TASK: submit_fournos_job: Submit the FOURNOS job
~~ ARTIFACT_DIR: /logs/artifacts/001__submit_and_wait
~~ LOG_FILE: /logs/artifacts/001__submit_and_wait/task.log
~~ ARGS:

[...]

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 10

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (2)
projects/core/dsl/runtime.py (1)

79-86: ⚠️ Potential issue | 🟠 Major

DSL metadata is still being written under _meta.

This block now writes task.log to the artifact root, but the same execute_tasks() call still creates meta_dir = env.ARTIFACT_DIR / "_meta" and passes it into _generate_execution_metadata() / _generate_restart_script(). After the rename in prepare_ci.py and config.py, a single run now spreads metadata across two directories.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/dsl/runtime.py` around lines 79 - 86, The code writes task.log
to the artifact root (log_file) but still generates metadata into meta_dir
(env.ARTIFACT_DIR / "_meta"), causing metadata to be split; change execute_tasks
so meta_dir is the same directory used by _setup_execution_logging/log_file
(e.g., set meta_dir = log_file.parent or use env.ARTIFACT_DIR consistently) and
pass that meta_dir into _generate_execution_metadata and
_generate_restart_script instead of the hardcoded "_meta" path; update
references to meta_dir in execute_tasks and any callers to ensure all metadata
and logs land in the same artifact directory.
projects/core/ci_entrypoint/prepare_ci.py (1)

382-407: ⚠️ Potential issue | 🟠 Major

Create 000__ci_metadata before the first write.

system_prechecks() now writes forge.git_version and both PR JSON files into a nested directory, but this function never creates it. prepare() calls this block before parse_and_save_pr_arguments(), so a fresh artifact dir will fail on Line 382 and both curl -o targets on Lines 399 and 407 point at a missing path.

🛠️ Suggested fix
     artifact_path = Path(artifact_dir)
+    metadata_dir = artifact_path / CI_METADATA_DIRNAME
+    metadata_dir.mkdir(parents=True, exist_ok=True)

     # Check for existing failures
     failures_file = artifact_path / "FAILURES"
@@
-        (artifact_path / CI_METADATA_DIRNAME / "forge.git_version").write_text(forge_version + "\n")
+        (metadata_dir / "forge.git_version").write_text(forge_version + "\n")
@@
-                ["curl", "-sSf", pr_url, "-o", str(artifact_path / CI_METADATA_DIRNAME / "pull_request.json")],
+                ["curl", "-sSf", pr_url, "-o", str(metadata_dir / "pull_request.json")],
@@
-                ["curl", "-sSf", pr_comments_url, "-o", str(artifact_path / CI_METADATA_DIRNAME / "pull_request-comments.json")],
+                ["curl", "-sSf", pr_comments_url, "-o", str(metadata_dir / "pull_request-comments.json")],
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/ci_entrypoint/prepare_ci.py` around lines 382 - 407,
system_prechecks() is writing files into a nested metadata dir (artifact_path /
CI_METADATA_DIRNAME) but never creates that directory; before writing
forge.git_version and before running the curl commands that output
pull_request.json and pull_request-comments.json, ensure the metadata directory
exists by calling Path.mkdir(parents=True, exist_ok=True) on artifact_path /
CI_METADATA_DIRNAME (or otherwise creating it). Add this creation at the start
of the block in system_prechecks() (or in prepare() if that guarantees the path)
so writes and subprocess curl -o targets succeed.
🧹 Nitpick comments (2)
projects/core/dsl/log.py (1)

29-29: Use explicit | None for optional type hints.

Static analysis flags that PEP 484 prohibits implicit Optional. Both function_args and log_file parameters should explicitly indicate they accept None.

♻️ Proposed fix for type hints
-def log_execution_banner(function_args: dict = None, log_file: str = None):
+def log_execution_banner(function_args: dict | None = None, log_file: str | None = None):
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/dsl/log.py` at line 29, Update the function signature for
log_execution_banner to use explicit optional type hints by changing the
parameters to function_args: dict | None = None and log_file: str | None = None
so both annotations explicitly allow None; locate and modify the def
log_execution_banner(...) declaration accordingly.
projects/core/ci_entrypoint/run_ci.py (1)

198-202: Consider providing a clearer error message when script exists but isn't executable.

The new executable check is appropriate, but when a script file exists but lacks execute permission, the error at line 396 will say "No CI script found" which may confuse users. Consider logging a warning or adjusting the error to clarify when the file exists but isn't executable.

💡 Suggested improvement for better diagnostics
     for script_path in possible_locations:
-        if script_path.exists() and os.access(script_path, os.X_OK):
+        if script_path.exists():
+            if os.access(script_path, os.X_OK):
+                return script_path
+            else:
+                logging.warning(f"Script exists but is not executable: {script_path}")
-            return script_path

     return None
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/ci_entrypoint/run_ci.py` around lines 198 - 202, The lookup
currently returns None when no executable is found causing a misleading "No CI
script found" error even if a candidate exists but lacks execute permission;
update the scan over possible_locations to detect and record any paths that
exist but are not executable (use the existing possible_locations variable and
the same existence/os.access checks), emit a clear warning listing those
non-executable files (via the module logger or existing logging facility), and
adjust the later error path (the code that currently raises "No CI script found"
at the caller site) to include a note that some scripts were found but not
executable so users know to check permissions.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@projects/core/dsl/runtime.py`:
- Around line 106-108: The current except TaskExecutionError: raise in
execute_tasks() short-circuits the always-task block; instead catch
TaskExecutionError, assign it to the existing execution_error variable (or
preserve the original exception), allow the function to continue so the `@always`
tasks block executes, and only after running the always-task and cleanup block
re-raise execution_error (i.e., keep the existing if execution_error: raise
execution_error logic so TaskExecutionError is raised after always-tasks
finish). Ensure you reference and update the exception handling around
execute_tasks(), TaskExecutionError, and the execution_error variable
accordingly.
- Around line 179-192: The relative_to(env.FORGE_HOME) call can raise ValueError
for files outside FORGE_HOME and mask the original exception; wrap the
Path(task_func.original_func.__code__.co_filename).relative_to(env.FORGE_HOME)
call in a try/except that catches ValueError and falls back to using the
absolute path (e.g., str(Path(...).resolve() or Path(...).as_posix())) when
computing task_location, then construct TaskExecutionError (task_name,
task_description, original_exception=e, task_args=...,
task_location=task_location, artifact_dir=...) and re-raise it so the original
stack/exception is preserved.

In `@projects/core/dsl/shell.py`:
- Around line 98-109: The enhanced message you build is lost because
CalledProcessError's str() ignores a modified args tuple and callers/logging at
lines where logger.error(f"<{e.__class__.__name__}> {e}") is used will show the
standard message; instead, before raising leave the exception structure intact
and explicitly log the enriched message: construct error_msg as you do, call
logger.error(error_msg) (or logger.exception if you want stack), then raise
subprocess.CalledProcessError(result.returncode, command, result.stdout,
result.stderr) without mutating error.args; reference the existing variables
check, result, command and the subprocess.CalledProcessError creation in this
block to locate and update the code.

In `@projects/core/dsl/toolbox.py`:
- Around line 109-131: The failure formatter get_task_execution_error currently
contains an interactive debugger call (import pdb; pdb.set_trace()) which will
hang all TaskExecutionError flows; remove the pdb import/call from
get_task_execution_error so the generator yields the summary and continues to
print/return the error info normally (ensure no leftover pdb references remain
in that function).

In `@projects/fournos_launcher/orchestration/submit.py`:
- Line 34: The logging call uses an unnecessary f-string: change the
logger.warning(f"Hello Fournos") in submit.py (the logger.warning call) to a
regular string without the f-prefix so it reads logger.warning("Hello Fournos");
keep the same message and logger variable, only remove the leading f.
- Around line 38-44: The call to config.project.get_config("fournos.namespace")
can raise KeyError if the key is missing; update the submit_and_wait invocation
(the call to submit_and_wait) to pass a safe default for namespace by retrieving
the config with a default (e.g., use the get_config overload or helper that
accepts a default value, or wrap the lookup in a try/except and fall back to
"fournos-jobs"), so that namespace defaults to "fournos-jobs" when the config
key is absent and submit_and_wait's own default behavior is preserved.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py`:
- Around line 186-214: The retrieve_job_logs and capture_final_job_status tasks
are not executed if wait_for_job_completion() raises, so move their invocation
into a finally block (or convert them to the framework's "always-run"
task/decorator) so they always run on failure or success; specifically, ensure
after calling wait_for_job_completion() (or in the try/except around it) you
call retrieve_job_logs(args) and capture_final_job_status(args) from a finally
clause (or replace `@task` with the project's always-run decorator) so the final
YAML and pod logs are captured even when wait_for_job_completion() throws.
- Around line 25-26: The run() function currently ignores timeout_minutes and
poll_interval_seconds because it uses a fixed `@retry`(attempts=120, delay=30)
decorator; change the logic so the retry behavior is driven by the passed-in
timeout_minutes and poll_interval_seconds (either by computing attempts =
ceil(timeout_minutes*60 / poll_interval_seconds) and using those values when
calling the retry helper or by replacing the decorator with an explicit loop
that sleeps poll_interval_seconds between polls and aborts after
timeout_minutes). Update the code paths that mirror this behavior (the other
occurrence around the second run/usage) so both places use the computed
attempts/delay or the explicit loop, and ensure variable names timeout_minutes
and poll_interval_seconds are actually referenced instead of hard-coded
constants.
- Around line 217-223: Swap the decorators on cleanup_job_manifest so `@task`
appears above `@always` (i.e., `@task` on the line immediately above the def and
`@always` just above it) so that the `@always` decorator is applied before the `@task`
registration and can set the _always_execute flag prior to task registration.
- Around line 79-86: The job name generation in generate_job_name currently
injects args.project (and any user-provided args.job_name) verbatim into
args.final_job_name which can violate Kubernetes DNS-1123 label rules; update
generate_job_name to normalize the name for both branches: convert to lowercase,
replace any character not in [a-z0-9.-] (e.g., underscores, spaces) with
hyphens, collapse consecutive hyphens, strip leading/trailing hyphens, and
truncate the final label to 63 characters before assigning to
args.final_job_name so both generated names and user overrides are always
DNS-1123 compliant.

---

Outside diff comments:
In `@projects/core/ci_entrypoint/prepare_ci.py`:
- Around line 382-407: system_prechecks() is writing files into a nested
metadata dir (artifact_path / CI_METADATA_DIRNAME) but never creates that
directory; before writing forge.git_version and before running the curl commands
that output pull_request.json and pull_request-comments.json, ensure the
metadata directory exists by calling Path.mkdir(parents=True, exist_ok=True) on
artifact_path / CI_METADATA_DIRNAME (or otherwise creating it). Add this
creation at the start of the block in system_prechecks() (or in prepare() if
that guarantees the path) so writes and subprocess curl -o targets succeed.

In `@projects/core/dsl/runtime.py`:
- Around line 79-86: The code writes task.log to the artifact root (log_file)
but still generates metadata into meta_dir (env.ARTIFACT_DIR / "_meta"), causing
metadata to be split; change execute_tasks so meta_dir is the same directory
used by _setup_execution_logging/log_file (e.g., set meta_dir = log_file.parent
or use env.ARTIFACT_DIR consistently) and pass that meta_dir into
_generate_execution_metadata and _generate_restart_script instead of the
hardcoded "_meta" path; update references to meta_dir in execute_tasks and any
callers to ensure all metadata and logs land in the same artifact directory.

---

Nitpick comments:
In `@projects/core/ci_entrypoint/run_ci.py`:
- Around line 198-202: The lookup currently returns None when no executable is
found causing a misleading "No CI script found" error even if a candidate exists
but lacks execute permission; update the scan over possible_locations to detect
and record any paths that exist but are not executable (use the existing
possible_locations variable and the same existence/os.access checks), emit a
clear warning listing those non-executable files (via the module logger or
existing logging facility), and adjust the later error path (the code that
currently raises "No CI script found" at the caller site) to include a note that
some scripts were found but not executable so users know to check permissions.

In `@projects/core/dsl/log.py`:
- Line 29: Update the function signature for log_execution_banner to use
explicit optional type hints by changing the parameters to function_args: dict |
None = None and log_file: str | None = None so both annotations explicitly allow
None; locate and modify the def log_execution_banner(...) declaration
accordingly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ce209994-e68a-41df-98a7-a9cae336347b

📥 Commits

Reviewing files that changed from the base of the PR and between 210aa00 and df63e25.

📒 Files selected for processing (17)
  • projects/core/ci_entrypoint/prepare_ci.py
  • projects/core/ci_entrypoint/run_ci.py
  • projects/core/dsl/__init__.py
  • projects/core/dsl/log.py
  • projects/core/dsl/runtime.py
  • projects/core/dsl/shell.py
  • projects/core/dsl/task.py
  • projects/core/dsl/toolbox.py
  • projects/core/library/ci.py
  • projects/core/library/config.py
  • projects/core/library/env.py
  • projects/core/notifications/send.py
  • projects/fournos_launcher/orchestration/ci.py
  • projects/fournos_launcher/orchestration/cli.py
  • projects/fournos_launcher/orchestration/config.yaml
  • projects/fournos_launcher/orchestration/submit.py
  • projects/fournos_launcher/toolbox/submit_and_wait/main.py
💤 Files with no reviewable changes (1)
  • projects/core/library/env.py

Comment thread projects/core/dsl/runtime.py Outdated
Comment thread projects/core/dsl/runtime.py Outdated
Comment thread projects/core/dsl/shell.py
Comment thread projects/fournos_launcher/orchestration/submit.py Outdated
Comment thread projects/fournos_launcher/orchestration/submit.py
Comment thread projects/fournos_launcher/toolbox/submit_and_wait/main.py Outdated
Comment thread projects/fournos_launcher/toolbox/submit_and_wait/main.py
Comment thread projects/fournos_launcher/toolbox/submit_and_wait/main.py
Comment thread projects/fournos_launcher/toolbox/submit_and_wait/main.py
@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 06 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator:

## /logs/artifacts/FAILURE 
================================================================================
🚨 CI EXECUTION FAILED
================================================================================

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
~~ projects/fournos_launcher/toolbox/submit_and_wait/main.py:133
~~ TASK: submit_fournos_job: Submit the FOURNOS job
~~ ARTIFACT_DIR: /logs/artifacts/001__submit_and_wait
~~ LOG_FILE: /logs/artifacts/001__submit_and_wait/task.log

[...]

@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 05 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator:

## /logs/artifacts/FAILURE 
================================================================================
🚨 CI EXECUTION FAILED
================================================================================

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
~~ projects/fournos_launcher/toolbox/submit_and_wait/main.py:133
~~ TASK: submit_fournos_job: Submit the FOURNOS job
~~ ARTIFACT_DIR: /logs/artifacts/001__submit_and_wait
~~ LOG_FILE: /logs/artifacts/001__submit_and_wait/task.log

[...]

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

♻️ Duplicate comments (7)
projects/core/dsl/runtime.py (2)

179-192: ⚠️ Potential issue | 🟠 Major

Guard relative_to() so it can't mask the real task exception.

If a task function lives outside env.FORGE_HOME, Path.relative_to() raises ValueError while you're already handling another exception. The wrapper then replaces the original task failure with a location-formatting failure.

Safer location formatting
-        co_filename = Path(task_func.original_func.__code__.co_filename).relative_to(env.FORGE_HOME)
-        task_location = f"{co_filename}:{task_func.original_func.__code__.co_firstlineno}"
+        co_filename = Path(task_func.original_func.__code__.co_filename)
+        try:
+            co_filename = co_filename.relative_to(env.FORGE_HOME)
+        except ValueError:
+            pass
+        task_location = f"{co_filename}:{task_func.original_func.__code__.co_firstlineno}"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/dsl/runtime.py` around lines 179 - 192, The current
construction of task_location uses Path(...).relative_to(env.FORGE_HOME) which
can raise ValueError and mask the original exception; update the logic around
the location formatting (where task_location is built, referencing
task_func.original_func.__code__.co_filename and co_firstlineno) to catch
exceptions from Path.relative_to and fall back to using the absolute path (or
the original co_filename) when relative_to fails, ensuring any error from
relative path computation does not override the original exception before you
raise TaskExecutionError; preserve the original exception in the
TaskExecutionError's original_exception field.

106-108: ⚠️ Potential issue | 🔴 Critical

Don't short-circuit the always-task phase on task failures.

_execute_single_task() wraps regular task exceptions as TaskExecutionError, so this branch exits execute_tasks() before lines 122-136 run. That breaks failure-path cleanup like cleanup_job_manifest() in projects/fournos_launcher/toolbox/submit_and_wait/main.py:217-223.

Minimal fix
-        except TaskExecutionError:
-            # Re-raise the TaskExecutionError to preserve context
-            raise
+        except TaskExecutionError as e:
+            execution_error = e
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/dsl/runtime.py` around lines 106 - 108, The except
TaskExecutionError block in execute_tasks prematurely re-raises errors from
_execute_single_task and skips the always-task phase; change the handler to
capture the TaskExecutionError into a local variable (e.g., task_exc = exc)
without re-raising, continue execution so the always-task/cleanup block (the
code currently at lines ~122-136) runs, and after the always-task phase re-raise
the stored task_exc if set; reference execute_tasks and _execute_single_task and
ensure cleanup_job_manifest and other always-task logic always run before
re-raising.
projects/fournos_launcher/toolbox/submit_and_wait/main.py (4)

217-218: ⚠️ Potential issue | 🔴 Critical

Swap the decorators so cleanup is really always-run.

Python applies decorators bottom-up. With the current order, @task registers the function before @always marks it, so cleanup_job_manifest() is treated as a normal task and won't run on failures.

Minimal fix
-@always
-@task
+@task
+@always
 def cleanup_job_manifest(args):
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py` around lines 217 -
218, The decorators on cleanup_job_manifest are applied in the wrong order so
`@task` wraps the function before `@always`, preventing the cleanup from being
treated as an always-run action; swap the decorators so `@always` is the outer
decorator (i.e., apply `@task` first then `@always` in source order) on the
cleanup_job_manifest function to ensure it runs on failures.

24-26: ⚠️ Potential issue | 🟠 Major

timeout_minutes and poll_interval_seconds are no-ops right now.

The wait behavior is still hard-coded by @retry(attempts=120, delay=30), so callers always poll for 60 minutes at 30-second intervals. That diverges from both the function signature and the manifest timeout, which uses args.timeout_minutes.

Also applies to: 146-148

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py` around lines 24 -
26, The retry loop is hard-coded with `@retry`(attempts=120, delay=30) so
timeout_minutes and poll_interval_seconds are ignored; update the wait logic in
main.py (where the `@retry` decorator is applied to the submit/wait helper) to
derive attempts and delay from the function arguments (timeout_minutes and
poll_interval_seconds) or replace the decorator with an explicit polling loop
that uses attempts = ceil(timeout_minutes*60 / poll_interval_seconds) and
sleep(poll_interval_seconds), and ensure the same change is applied to the
duplicate block referenced around lines 146-148 so callers and the manifest's
args.timeout_minutes actually control the polling behavior.

186-214: ⚠️ Potential issue | 🟠 Major

Keep log/YAML capture on the failure path too.

Even after the runtime always-task flow is fixed, these are still normal tasks. If wait_for_job_completion() raises, the failure case skips the final status YAML and pod logs—the artifacts you usually need most to debug the job.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py` around lines 186 -
214, Ensure job logs and final-status YAML are always captured on failure by
invoking retrieve_job_logs and capture_final_job_status from the failure/cleanup
path of the orchestration that calls wait_for_job_completion — e.g., move their
invocation into an except/finally or a runtime-always cleanup block so they run
even if wait_for_job_completion raises; keep using their existing shell.run
calls (check=False) and the functions retrieve_job_logs and
capture_final_job_status names to locate and update the orchestration logic.

82-86: ⚠️ Potential issue | 🟠 Major

Normalize the job name before writing it into metadata.name.

The default path uses args.project verbatim, and both the docstring and PR usage examples use llm_d. That yields names like forge-llm_d-20260407-211544, which Kubernetes rejects because _ is not valid in a DNS-1123 resource name. The same normalization/validation needs to apply to custom job_name values.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py` around lines 82 -
86, Normalize any job name (both when assigning args.final_job_name from
args.job_name and when composing the default using args.project) to a DNS-1123
compliant string before writing it into metadata.name: implement or call a
helper (e.g., normalize_name) that lowercases the string, replaces invalid chars
like "_" with "-", removes/strips characters not in [a-z0-9-], collapses
consecutive "-"s, trims leading/trailing non-alphanumeric characters, and
enforces the 63-char max length; then use the normalized value for
args.final_job_name instead of the raw args.job_name or args.project.
projects/fournos_launcher/orchestration/submit.py (1)

38-44: ⚠️ Potential issue | 🟠 Major

Keep the toolbox default namespace reachable.

projects/fournos_launcher/toolbox/submit_and_wait/main.py:24 already defaults namespace to "fournos-jobs". This unconditional get_config("fournos.namespace") makes that fallback unreachable and turns a missing config key into a submit-time failure instead.

Minimal fix
     submit_and_wait(
       cluster_name=config.project.get_config("cluster.name"),
       project=config.project.get_config("ci_job.project"),
       args=config.project.get_config("ci_job.args"),
       variables_overrides=config.project.get_config("overrides"),
-      namespace=config.project.get_config("fournos.namespace"),
+      namespace=config.project.get_config("fournos.namespace", default_value="fournos-jobs"),
     )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/orchestration/submit.py` around lines 38 - 44, The
call to submit_and_wait unconditionally passes
namespace=config.project.get_config("fournos.namespace"), which overrides the
toolbox default ("fournos-jobs") and causes a failure when the config key is
missing; change the call so namespace is omitted when the config key is absent
or pass None/a fallback from get_config (e.g.,
namespace=config.project.get_config("fournos.namespace", default=None)) so
submit_and_wait (in main.py) can use its internal default; update the
submit_and_wait invocation accordingly and keep referencing the submit_and_wait
function and config.project.get_config to locate the change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@projects/core/dsl/log.py`:
- Line 29: The function log_execution_banner declares function_args: dict = None
but its body calls function_args.items(), causing a crash when callers omit the
argument; update log_execution_banner to normalize the parameter at start (e.g.,
set function_args = function_args or {}) so subsequent calls to
function_args.items() are safe, or alternatively make function_args a required
parameter by removing the default None—ensure references in the function (the
items() usage) and any callers are consistent with the chosen approach.

In `@projects/core/dsl/runtime.py`:
- Around line 79-82: A new FileHandler created by _setup_execution_logging is
never removed causing duplicate logs and leaked fds across multiple calls to
execute_tasks; modify _setup_execution_logging to return the created handler (or
otherwise expose it) and then in execute_tasks, after calling
log_execution_banner and when execution ends (use finally), call
logger.removeHandler(handler) and handler.close() to properly detach and close
the per-run handler; ensure the same pattern is applied where similar handlers
are created (the region around lines 236-251) so each execution scopes and
cleans up its FileHandler.

In `@projects/fournos_launcher/orchestration/submit.py`:
- Around line 33-34: The submit_job function currently logs a placeholder
warning ("Hello Fournos") which pollutes CI; replace this logger.warning call
with either removal or a meaningful lower-severity breadcrumb (logger.info or
logger.debug) and a concise message about the successful happy-path action
(e.g., "job submission started" or "job submitted successfully") so only real
warnings remain at warning level; update the logger call inside submit_job
accordingly.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py`:
- Around line 17-27: The CLI is overwriting user-supplied
--args/--variables-overrides because the function signature uses mutable
list/dict defaults; change the run signature to use None defaults (e.g., args:
Optional[list] = None, variables_overrides: Optional[dict] = None) and, at the
start of run (and the other signatures noted around lines 71-73), normalize them
into empty structures only when they are None (args = [] if args is None,
variables_overrides = {} if variables_overrides is None) so create_toolbox_main
/ the CLI type-fallback logic no longer erases CLI-provided values.

---

Duplicate comments:
In `@projects/core/dsl/runtime.py`:
- Around line 179-192: The current construction of task_location uses
Path(...).relative_to(env.FORGE_HOME) which can raise ValueError and mask the
original exception; update the logic around the location formatting (where
task_location is built, referencing task_func.original_func.__code__.co_filename
and co_firstlineno) to catch exceptions from Path.relative_to and fall back to
using the absolute path (or the original co_filename) when relative_to fails,
ensuring any error from relative path computation does not override the original
exception before you raise TaskExecutionError; preserve the original exception
in the TaskExecutionError's original_exception field.
- Around line 106-108: The except TaskExecutionError block in execute_tasks
prematurely re-raises errors from _execute_single_task and skips the always-task
phase; change the handler to capture the TaskExecutionError into a local
variable (e.g., task_exc = exc) without re-raising, continue execution so the
always-task/cleanup block (the code currently at lines ~122-136) runs, and after
the always-task phase re-raise the stored task_exc if set; reference
execute_tasks and _execute_single_task and ensure cleanup_job_manifest and other
always-task logic always run before re-raising.

In `@projects/fournos_launcher/orchestration/submit.py`:
- Around line 38-44: The call to submit_and_wait unconditionally passes
namespace=config.project.get_config("fournos.namespace"), which overrides the
toolbox default ("fournos-jobs") and causes a failure when the config key is
missing; change the call so namespace is omitted when the config key is absent
or pass None/a fallback from get_config (e.g.,
namespace=config.project.get_config("fournos.namespace", default=None)) so
submit_and_wait (in main.py) can use its internal default; update the
submit_and_wait invocation accordingly and keep referencing the submit_and_wait
function and config.project.get_config to locate the change.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py`:
- Around line 217-218: The decorators on cleanup_job_manifest are applied in the
wrong order so `@task` wraps the function before `@always`, preventing the cleanup
from being treated as an always-run action; swap the decorators so `@always` is
the outer decorator (i.e., apply `@task` first then `@always` in source order) on
the cleanup_job_manifest function to ensure it runs on failures.
- Around line 24-26: The retry loop is hard-coded with `@retry`(attempts=120,
delay=30) so timeout_minutes and poll_interval_seconds are ignored; update the
wait logic in main.py (where the `@retry` decorator is applied to the submit/wait
helper) to derive attempts and delay from the function arguments
(timeout_minutes and poll_interval_seconds) or replace the decorator with an
explicit polling loop that uses attempts = ceil(timeout_minutes*60 /
poll_interval_seconds) and sleep(poll_interval_seconds), and ensure the same
change is applied to the duplicate block referenced around lines 146-148 so
callers and the manifest's args.timeout_minutes actually control the polling
behavior.
- Around line 186-214: Ensure job logs and final-status YAML are always captured
on failure by invoking retrieve_job_logs and capture_final_job_status from the
failure/cleanup path of the orchestration that calls wait_for_job_completion —
e.g., move their invocation into an except/finally or a runtime-always cleanup
block so they run even if wait_for_job_completion raises; keep using their
existing shell.run calls (check=False) and the functions retrieve_job_logs and
capture_final_job_status names to locate and update the orchestration logic.
- Around line 82-86: Normalize any job name (both when assigning
args.final_job_name from args.job_name and when composing the default using
args.project) to a DNS-1123 compliant string before writing it into
metadata.name: implement or call a helper (e.g., normalize_name) that lowercases
the string, replaces invalid chars like "_" with "-", removes/strips characters
not in [a-z0-9-], collapses consecutive "-"s, trims leading/trailing
non-alphanumeric characters, and enforces the 63-char max length; then use the
normalized value for args.final_job_name instead of the raw args.job_name or
args.project.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5506ef61-f13e-4d20-a8c5-623b5517df20

📥 Commits

Reviewing files that changed from the base of the PR and between df63e25 and db73265.

📒 Files selected for processing (13)
  • projects/core/dsl/__init__.py
  • projects/core/dsl/log.py
  • projects/core/dsl/runtime.py
  • projects/core/dsl/shell.py
  • projects/core/dsl/task.py
  • projects/core/dsl/toolbox.py
  • projects/core/library/ci.py
  • projects/core/library/env.py
  • projects/fournos_launcher/orchestration/ci.py
  • projects/fournos_launcher/orchestration/cli.py
  • projects/fournos_launcher/orchestration/config.yaml
  • projects/fournos_launcher/orchestration/submit.py
  • projects/fournos_launcher/toolbox/submit_and_wait/main.py
💤 Files with no reviewable changes (1)
  • projects/core/library/env.py
✅ Files skipped from review due to trivial changes (2)
  • projects/core/dsl/shell.py
  • projects/fournos_launcher/orchestration/config.yaml
🚧 Files skipped from review as they are similar to previous changes (4)
  • projects/fournos_launcher/orchestration/cli.py
  • projects/core/dsl/init.py
  • projects/fournos_launcher/orchestration/ci.py
  • projects/core/dsl/task.py

Comment thread projects/core/dsl/log.py
Comment thread projects/core/dsl/runtime.py Outdated
Comment thread projects/fournos_launcher/orchestration/submit.py Outdated
Comment thread projects/fournos_launcher/toolbox/submit_and_wait/main.py
@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 02 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator:

## /logs/artifacts/FAILURE 
================================================================================
🚨 CI EXECUTION FAILED
================================================================================

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
~~ projects/fournos_launcher/toolbox/submit_and_wait/main.py:133
~~ TASK: submit_fournos_job: Submit the FOURNOS job
~~ ARTIFACT_DIR: /logs/artifacts/001__submit_and_wait
~~ LOG_FILE: /logs/artifacts/001__submit_and_wait/task.log

[...]

@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 02 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator:

## /logs/artifacts/FAILURE 
================================================================================
🚨 CI EXECUTION FAILED
================================================================================

xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
~~ projects/fournos_launcher/toolbox/submit_and_wait/main.py:133
~~ TASK: submit_fournos_job: Submit the FOURNOS job
~~ ARTIFACT_DIR: /logs/artifacts/001__submit_and_wait
~~ LOG_FILE: /logs/artifacts/001__submit_and_wait/task.log
~~ ARGS:
~~     cluster_name: psap
~~     project: llm_d
~~     args: &id001
~~     - hello
~~     - world
~~     variables_overrides: &id002
~~       something.is.wrong: 'false'
~~     job_name: ''
~~     namespace: psap-automation
~~     timeout_minutes: 120
~~     poll_interval_seconds: 30
~~     artifact_dir: /logs/artifacts/001__submit_and_wait
~~     args_list: *id001
~~     variables_dict: *id002
~~     final_job_name: forge-llm_d-20260407-214700
~~     manifest_file: /logs/artifacts/001__submit_and_wait/src/forge-llm_d-20260407-214700-manifest.yaml
~~ EXCEPTION: CalledProcessError
~~     Command 'oc apply -f /logs/artifacts/001__submit_and_wait/src/forge-llm_d-20260407-214700-manifest.yaml' returned non-zero exit status 1.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


[...]

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
projects/core/library/ci.py (1)

12-15: Remove unused imports.

The imports inspect, os, and yaml (lines 12-14) are not used anywhere in the file. Consider removing them.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/library/ci.py` around lines 12 - 15, The top-level imports
include unused symbols inspect, os, and yaml; remove those import statements and
keep only the needed import (Path) so the module no longer imports unused names.
Update the import block to eliminate inspect, os, and yaml references and run a
quick lint/test to ensure no remaining uses of those symbols elsewhere in the
module (check for usages of inspect, os, or yaml and remove or replace them if
found).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@projects/core/ci_entrypoint/prepare_ci.py`:
- Around line 382-383: system_prechecks() writes artifact_path /
CI_METADATA_DIRNAME / "forge.git_version" but the CI_METADATA_DIRNAME directory
may not exist if parse_and_save_pr_arguments() hasn't run; modify
system_prechecks() (or the prepare() flow) to ensure the directory exists before
writing by creating artifact_path / CI_METADATA_DIRNAME (use a mkdir with
parents=True and exist_ok=True) prior to the write operation so the write can
never raise FileNotFoundError.

In `@projects/core/notifications/send.py`:
- Around line 173-184: The code uses lines.index("---\n") to find the stacktrace
delimiter and then only displays lines before it, which truncates summaries when
multiple FAILURE entries are aggregated; change the search to only look for the
delimiter inside the initial summary window (use DEFAULT_HEAD) e.g. scan
lines[:DEFAULT_HEAD] for the first occurrence instead of calling
lines.index("---\n") on the whole file so head = next((i for i,l in
enumerate(lines[:DEFAULT_HEAD]) if l == "---\n"), DEFAULT_HEAD); keep the rest
of the message construction (message, get_link) unchanged so you still show the
intended summary block without cutting off later failures.

---

Nitpick comments:
In `@projects/core/library/ci.py`:
- Around line 12-15: The top-level imports include unused symbols inspect, os,
and yaml; remove those import statements and keep only the needed import (Path)
so the module no longer imports unused names. Update the import block to
eliminate inspect, os, and yaml references and run a quick lint/test to ensure
no remaining uses of those symbols elsewhere in the module (check for usages of
inspect, os, or yaml and remove or replace them if found).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 98d10328-3e78-4db1-9a18-fb81fc1f1816

📥 Commits

Reviewing files that changed from the base of the PR and between db73265 and 9f2834c.

📒 Files selected for processing (4)
  • projects/core/ci_entrypoint/prepare_ci.py
  • projects/core/library/ci.py
  • projects/core/library/env.py
  • projects/core/notifications/send.py
💤 Files with no reviewable changes (1)
  • projects/core/library/env.py

Comment thread projects/core/ci_entrypoint/prepare_ci.py
Comment thread projects/core/notifications/send.py
@kpouget

kpouget commented Apr 7, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 7, 2026

Copy link
Copy Markdown

🔴 Test of 'fournos_launcher submit' failed after 00 hours 00 minutes 03 seconds 🔴

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

Failure indicator: Empty. (See run.log)

@kpouget

kpouget commented Apr 8, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
projects/core/dsl/task.py (1)

231-291: ⚠️ Potential issue | 🔴 Critical

@when / @always / @retry no longer affect the registered task.

With the new contract, these decorators run after @task, but task() has already snapshotted condition / always_execute and stored the inner task wrapper in _task_registry. The updated flags never reach task_info, and @retry wraps a function that execute_tasks() never calls. In this PR, wait_for_job_completion() will execute once and cleanup_job_manifest() will not be treated as an always task.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/dsl/task.py` around lines 231 - 291, Decorators `@when`, `@always`
and retry currently set attributes on the original function but run after `@task`
has already registered a wrapped task in _task_registry, so those flags never
reach task_info and retry wraps the wrong callable; update these decorators
(when, always, retry) to after setting func._... also locate the
already-registered task wrapper in _task_registry (using the function
identity/name used by task) and apply the changes to that registered entry:
assign the condition to task_info.condition, set task_info.always_execute = True
for always, and wrap task_info.func with the _execute_with_retry wrapper (or
update task_info._retry_config) for retry so execute_tasks() and functions like
wait_for_job_completion / cleanup_job_manifest see the updated behavior. Ensure
you reference and modify the same task_info/_task_registry entries that task()
created rather than only mutating the original inner function.
♻️ Duplicate comments (5)
projects/core/dsl/runtime.py (2)

183-198: ⚠️ Potential issue | 🟠 Major

Don't shadow the original task exception in the path fallback.

The inner except ValueError as e overwrites the outer task exception. If relative_to() fails, that inner handler clears the binding, and original_exception=e can raise UnboundLocalError instead of wrapping the real task error.

Minimal fix
-        except ValueError as e:
-            logging.warning(f"Path {co_filename} isn't relative to FORGE_HOME={env.FORGE_HOME} ({e})")
+        except ValueError as path_error:
+            logging.warning(f"Path {co_filename} isn't relative to FORGE_HOME={env.FORGE_HOME} ({path_error})")
             pass  # Use absolute path if file is outside FORGE_HOME
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/dsl/runtime.py` around lines 183 - 198, The inner except
currently uses "except ValueError as e" which shadows the outer task exception
variable so TaskExecutionError(original_exception=e) may reference the
wrong/undefined e; change the inner exception variable to a distinct name (e.g.
path_err) or avoid binding it, and ensure TaskExecutionError.original_exception
is assigned the outer task exception variable (the original task error captured
earlier), referencing symbols like co_filename, env.FORGE_HOME, the except
ValueError handler, TaskExecutionError, and
task_func.original_func.__code__.co_firstlineno so the path fallback no longer
overwrites the real task exception.

107-117: ⚠️ Potential issue | 🔴 Critical

Re-raise after the always_tasks block, not before it.

This still exits execute_tasks() immediately for TaskExecutionError, ConditionError, and generic task failures. _execute_single_task() wraps ordinary task exceptions as TaskExecutionError, so the common failure path skips every @always task.

Minimal fix
-                if isinstance(e, RetryFailure):
-                    # catch and execute the ALWAYS tasks
-                    execution_error = e
-                else:
-                    raise
+                execution_error = e

Also applies to: 119-133

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/core/dsl/runtime.py` around lines 107 - 117, The current except
block in execute_tasks() logs and immediately re-raises
TaskExecutionError/ConditionError/Exception which prevents any `@always` tasks
from running; change the flow so that when catching (TaskExecutionError,
ConditionError, RetryFailure, Exception) you assign execution_error = e (for all
exceptions), log the error (logger.fatal + log_completion_banner) but do NOT
raise there, then after running the always-tasks block (the code that executes
`@always` tasks) re-raise the original exception unless it was intended to be
handled as a RetryFailure; reference execute_tasks(), _execute_single_task(),
RetryFailure, TaskExecutionError, ConditionError, log_completion_banner and
execution_error when making this change.
projects/fournos_launcher/toolbox/submit_and_wait/main.py (3)

181-209: ⚠️ Potential issue | 🟠 Major

Keep final status and log collection on the failure path.

These are regular tasks after wait_for_job_completion(). Any exception from the wait step stops the run before either artifact is captured, which is exactly when the final YAML and pod logs are most valuable. Move them into a finally path or make them always-run tasks once the DSL always-task regression is fixed.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py` around lines 181 -
209, The final job logs and YAML are only collected by retrieve_job_logs and
capture_final_job_status when wait_for_job_completion completes successfully;
move or invoke these tasks so they always run on the failure path—wrap the call
to wait_for_job_completion in a try/finally and call retrieve_job_logs(args) and
capture_final_job_status(args) from the finally block (or mark them as
always-run tasks when the DSL regression is resolved) so that the artifacts are
captured even if wait_for_job_completion raises.

78-82: ⚠️ Potential issue | 🟠 Major

Normalize final_job_name before writing metadata.name.

Both branches accept raw project / job_name. Names like the documented llm_d example produce forge-llm_d-..., which Kubernetes rejects for resource names, so oc apply fails before the job is created. The same normalization/length checks need to apply to custom job_name values too.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py` around lines 78 -
82, Normalize and validate args.final_job_name before writing metadata.name:
apply the same sanitization used for generated names to both branches (when
setting from args.job_name or when composing
f"forge-{args.project}-{timestamp}"): convert to lowercase, replace invalid
chars (e.g., underscores, spaces) with '-', trim to Kubernetes max length (63
chars), and ensure it starts and ends with an alphanumeric character (adjust by
truncation or removing leading/trailing non-alphanumerics). Update the
assignment to args.final_job_name in main.py so the sanitization function (or
inline steps) is applied to args.job_name and to the composed
forge-{project}-{timestamp} value to prevent oc apply failures.

67-69: ⚠️ Potential issue | 🟠 Major

Don't coerce CLI-supplied structured inputs back to empty values.

create_dynamic_parser() currently parses these annotations as str, not list / dict (projects/core/dsl/cli.py:42-144). These isinstance(..., list/dict) guards therefore discard any toolbox --args / --variables-overrides input and submit the job without them. Parse string input here (for example YAML/JSON) or teach the parser how to materialize structured values.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py` around lines 67 -
69, The current guards in the main.py assignments (args.args_list and
args.variables_dict) drop CLI-provided structured values because
create_dynamic_parser() yields strings; update the handling in
submit_and_wait/main.py to deserialize string inputs into structured types
before coercion: detect if args.args is a str and parse it (e.g., YAML/JSON via
a safe loader) and validate it results in a list before assigning to
args.args_list, and similarly detect/parse args.variables_overrides into a dict
before assigning to args.variables_dict; reference create_dynamic_parser()
(projects/core/dsl/cli.py) as the origin of the string inputs and ensure the
parsing is robust to already-materialized list/dict values so existing list/dict
instances remain unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@projects/core/dsl/task.py`:
- Around line 131-148: The wrapper currently treats any single callable arg as
the decorated function, which misclassifies `@when`(lambda: ...) as a direct
decoration; change wrapper to decide simple-decorator vs decorator-factory using
the signature of decorator_func instead of callable(args[0]). Use
inspect.signature(decorator_func) to detect whether decorator_func is a plain
decorator that expects a single function argument (then run the simple-decorator
branch: set func/name, call _ensure_is_task and return decorator_func(func));
otherwise always return inner_decorator that calls decorator_func(*args,
**kwargs) and applies the resulting decorator to func. Update the branches
around wrapper, decorator_func, _ensure_is_task, and inner_decorator
accordingly.

---

Outside diff comments:
In `@projects/core/dsl/task.py`:
- Around line 231-291: Decorators `@when`, `@always` and retry currently set
attributes on the original function but run after `@task` has already registered a
wrapped task in _task_registry, so those flags never reach task_info and retry
wraps the wrong callable; update these decorators (when, always, retry) to after
setting func._... also locate the already-registered task wrapper in
_task_registry (using the function identity/name used by task) and apply the
changes to that registered entry: assign the condition to task_info.condition,
set task_info.always_execute = True for always, and wrap task_info.func with the
_execute_with_retry wrapper (or update task_info._retry_config) for retry so
execute_tasks() and functions like wait_for_job_completion /
cleanup_job_manifest see the updated behavior. Ensure you reference and modify
the same task_info/_task_registry entries that task() created rather than only
mutating the original inner function.

---

Duplicate comments:
In `@projects/core/dsl/runtime.py`:
- Around line 183-198: The inner except currently uses "except ValueError as e"
which shadows the outer task exception variable so
TaskExecutionError(original_exception=e) may reference the wrong/undefined e;
change the inner exception variable to a distinct name (e.g. path_err) or avoid
binding it, and ensure TaskExecutionError.original_exception is assigned the
outer task exception variable (the original task error captured earlier),
referencing symbols like co_filename, env.FORGE_HOME, the except ValueError
handler, TaskExecutionError, and task_func.original_func.__code__.co_firstlineno
so the path fallback no longer overwrites the real task exception.
- Around line 107-117: The current except block in execute_tasks() logs and
immediately re-raises TaskExecutionError/ConditionError/Exception which prevents
any `@always` tasks from running; change the flow so that when catching
(TaskExecutionError, ConditionError, RetryFailure, Exception) you assign
execution_error = e (for all exceptions), log the error (logger.fatal +
log_completion_banner) but do NOT raise there, then after running the
always-tasks block (the code that executes `@always` tasks) re-raise the original
exception unless it was intended to be handled as a RetryFailure; reference
execute_tasks(), _execute_single_task(), RetryFailure, TaskExecutionError,
ConditionError, log_completion_banner and execution_error when making this
change.

In `@projects/fournos_launcher/toolbox/submit_and_wait/main.py`:
- Around line 181-209: The final job logs and YAML are only collected by
retrieve_job_logs and capture_final_job_status when wait_for_job_completion
completes successfully; move or invoke these tasks so they always run on the
failure path—wrap the call to wait_for_job_completion in a try/finally and call
retrieve_job_logs(args) and capture_final_job_status(args) from the finally
block (or mark them as always-run tasks when the DSL regression is resolved) so
that the artifacts are captured even if wait_for_job_completion raises.
- Around line 78-82: Normalize and validate args.final_job_name before writing
metadata.name: apply the same sanitization used for generated names to both
branches (when setting from args.job_name or when composing
f"forge-{args.project}-{timestamp}"): convert to lowercase, replace invalid
chars (e.g., underscores, spaces) with '-', trim to Kubernetes max length (63
chars), and ensure it starts and ends with an alphanumeric character (adjust by
truncation or removing leading/trailing non-alphanumerics). Update the
assignment to args.final_job_name in main.py so the sanitization function (or
inline steps) is applied to args.job_name and to the composed
forge-{project}-{timestamp} value to prevent oc apply failures.
- Around line 67-69: The current guards in the main.py assignments
(args.args_list and args.variables_dict) drop CLI-provided structured values
because create_dynamic_parser() yields strings; update the handling in
submit_and_wait/main.py to deserialize string inputs into structured types
before coercion: detect if args.args is a str and parse it (e.g., YAML/JSON via
a safe loader) and validate it results in a list before assigning to
args.args_list, and similarly detect/parse args.variables_overrides into a dict
before assigning to args.variables_dict; reference create_dynamic_parser()
(projects/core/dsl/cli.py) as the origin of the string inputs and ensure the
parsing is robust to already-materialized list/dict values so existing list/dict
instances remain unchanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: ddd28897-13f2-4bdc-b7c2-e51ca882d0c7

📥 Commits

Reviewing files that changed from the base of the PR and between acc1c2f and 34222fe.

📒 Files selected for processing (16)
  • projects/core/ci_entrypoint/prepare_ci.py
  • projects/core/dsl/__init__.py
  • projects/core/dsl/log.py
  • projects/core/dsl/runtime.py
  • projects/core/dsl/shell.py
  • projects/core/dsl/task.py
  • projects/core/dsl/toolbox.py
  • projects/core/library/ci.py
  • projects/core/library/config.py
  • projects/core/library/env.py
  • projects/core/notifications/send.py
  • projects/fournos_launcher/orchestration/ci.py
  • projects/fournos_launcher/orchestration/cli.py
  • projects/fournos_launcher/orchestration/config.yaml
  • projects/fournos_launcher/orchestration/submit.py
  • projects/fournos_launcher/toolbox/submit_and_wait/main.py
💤 Files with no reviewable changes (1)
  • projects/core/library/env.py
✅ Files skipped from review due to trivial changes (3)
  • projects/core/dsl/init.py
  • projects/core/notifications/send.py
  • projects/fournos_launcher/orchestration/config.yaml
🚧 Files skipped from review as they are similar to previous changes (5)
  • projects/fournos_launcher/orchestration/cli.py
  • projects/fournos_launcher/orchestration/ci.py
  • projects/core/dsl/shell.py
  • projects/fournos_launcher/orchestration/submit.py
  • projects/core/library/config.py

Comment thread projects/core/dsl/task.py
@kpouget

kpouget commented Apr 8, 2026

Copy link
Copy Markdown
Contributor Author

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

@psap-forge-bot

psap-forge-bot Bot commented Apr 8, 2026

Copy link
Copy Markdown

• Link to the test results.

• No reports index generated...

Test configuration:

/test fournos llm_d hello world
/cluster psap
/var something.is.wrong: false

• Failure indicator*:

## /logs/artifacts/FAILURE 
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
~~ projects/fournos_launcher/toolbox/submit_and_wait/main.py:128
~~ TASK: submit_fournos_job: Submit the FOURNOS job
~~ ARTIFACT_DIR: /logs/artifacts/001__submit_and_wait
~~ LOG_FILE: /logs/artifacts/001__submit_and_wait/task.log
~~ ARGS:
~~     cluster_name: psap
~~     project: llm_d
~~     args: &id001
~~     - hello
~~     - world
~~     variables_overrides: &id002
~~       something.is.wrong: 'false'
~~     job_name: ''
~~     namespace: psap-automation
~~     artifact_dir: /logs/artifacts/001__submit_and_wait
~~     args_list: *id001
~~     variables_dict: *id002
~~     final_job_name: forge-llm_d-20260408-082243
~~     manifest_file: /logs/artifacts/001__submit_and_wait/src/forge-llm_d-20260408-082243-manifest.yaml
~~
~~ EXCEPTION: CalledProcessError
~~     Command 'oc apply -f /logs/artifacts/001__submit_and_wait/src/forge-llm_d-20260408-082243-manifest.yaml' returned non-zero exit status 1.
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx


[...]

• Execution logs

@openshift-ci

openshift-ci Bot commented Apr 8, 2026

Copy link
Copy Markdown

@kpouget: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/fournos 0821244 link true /test fournos

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@kpouget kpouget changed the title [fournos launcher] Make it work and improve the toolbox [fournos launcher] Improve the toolbox Apr 8, 2026
@kpouget

kpouget commented Apr 8, 2026

Copy link
Copy Markdown
Contributor Author

merging this version 👍🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant