Skip to content

ci: (PR 1/2) add the hermetic CPU contract suite - #1440

Merged
Luodian merged 3 commits into
mainfrom
brian/ci/hermetic-eval-contracts
Sep 5, 2026
Merged

ci: (PR 1/2) add the hermetic CPU contract suite#1440
Luodian merged 3 commits into
mainfrom
brian/ci/hermetic-eval-contracts

Conversation

@Luodian

@Luodian Luodian commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

We need a repeatable CPU check before changing the evaluation pipeline. This adds a pinned Linux environment, fixes test collection, and exercises the production request constructors without downloading datasets.

Scope

  • Collect the full test tree and run the CLI, registry, protocol, request, task, evaluator, and subprocess contracts offline.
  • Mark the Qwen2.5-VL integration test as GPU-only and supply the dependencies needed during collection.
  • Match the current Ruff configuration in the pinned requirements and test imports.

Validation

Rebased onto main at e0bfc699 on September 5, 2026.

  • Full collection: 680 tests, no collection errors in the local environment.
  • The workflow's seven-file CPU contract command: 177 passed locally.
  • uv run --no-sync pre-commit run --all-files: passed.
  • The pinned Python 3.11 Linux environment is validated by the Hermetic CPU Contracts workflow on this PR.

This adds the check but does not change branch protection. Requiring it is a separate repository setting.

Dependencies

Foundation stack: #1440 -> #1451. The sample-limit fix in #1441 also depends on this PR. This PR introduces no new evaluation lifecycle or model-reuse behavior.

@Luodian
Luodian force-pushed the brian/ci/hermetic-eval-contracts branch from b42c3c0 to 2b6ef4b Compare September 5, 2026 07:10
@Luodian
Luodian marked this pull request as ready for review September 5, 2026 07:11
@Luodian
Luodian merged commit 6b058d5 into main Sep 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant