Skip to content

fix(sdk): parse BashJob reward results (#1214)#1215

Draft
xdlkc wants to merge 5 commits into
alibaba:masterfrom
xdlkc:feature/bashjob-reward-result
Draft

fix(sdk): parse BashJob reward results (#1214)#1215
xdlkc wants to merge 5 commits into
alibaba:masterfrom
xdlkc:feature/bashjob-reward-result

Conversation

@xdlkc

@xdlkc xdlkc commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

fixes #1214

Summary

  • Add shared reward-protocol result models under rock.sdk.reward.result
  • Keep rock.sdk.job.result and Harbor result import paths backward-compatible via re-exports / thin wrappers
  • Teach BashTrial.collect() to discover <trial_name>/result.json under Bash job artifact roots and parse verifier_result.rewards.reward as trial.score
  • Propagate Bash process exit_code onto parsed reward-protocol trial results
  • Set a default LOG_DIR for BashJob sessions and keep stdout score: parsing as a compatibility fallback when no trial result file exists
  • Document the adaptation plan in docs/proposals/bashjob-reward-result-parsing.md
  • Document the Job result protocol in docs/dev/job/result-protocol.md

Root cause

BashJob collection only converted process exit status into the base TrialResult. The base result type has no verifier rewards, so TrialResult.score remains 0.0 even when the Bash job produced reward protocol artifacts.

Validation

  • uv run --extra admin --group test pytest tests/unit/sdk/job tests/unit/sdk/agent/test_job.py tests/unit/sdk/agent/test_models.py -q - 296 passed
  • uv run ruff check rock/sdk/job rock/sdk/bench rock/sdk/reward rock/sdk/result.py tests/unit/sdk/job/test_result.py tests/unit/sdk/agent/test_job.py tests/unit/sdk/agent/test_models.py
  • uv run ruff format --check rock/sdk/job rock/sdk/bench rock/sdk/reward rock/sdk/result.py tests/unit/sdk/job/test_result.py tests/unit/sdk/agent/test_job.py tests/unit/sdk/agent/test_models.py
  • import smoke for rock.sdk.reward.result, rock.sdk.job.result, and rock.sdk.bench.models.trial.result compatibility

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] BashJob SDK ignores reward protocol trial results

1 participant