Support external harness by sethkarten · Pull Request #1985 · PrimeIntellect-ai/verifiers

sethkarten · 2026-07-13T18:46:43Z

Summary

support installed top-level taskset, harness, and judge modules after built-in namespace lookup
preserve internal plugin import errors while falling back only when the candidate module or one of its parents is absent
document the external harness packaging ABI, ID normalization, __all__ export rule, and specialized config loading
test an actual built and installed external distribution through full EvalConfig and Environment construction
make successful Prime and Modal stop_confirmed() calls idempotent while preserving fail-closed missing-handle states and retryability after provider failure
exercise top-level judge config narrowing and construction from the same installed external plugin fixture

This stays generic. It contains no Emulator Bench or private harness implementation.

Validation

uv sync --frozen
targeted external-loader tests on Python 3.11, 3.12, and 3.13: 3 passed on each
non-v1 suite excluding environment installs and Prime integration: 851 passed
v1 suite excluding Prime and Modal with model keys unset: 66 passed, 60 skipped for missing PRIME_API_KEY
uv run --no-sync ruff check .
uv run --no-sync ruff format --check .
uv run --no-sync ty check verifiers
latest focused lifecycle/loader suite: 12 passed
latest v1 suite excluding live Prime and Modal cases: 75 passed, expected key-gated cases skipped
exact reviewed delta commit d6d52d7c418cbd297fcdfa09e1779a6586d29814, patch SHA-256 c7d84fab007e74c08d71dbbcdbefb1c12b5b2402991869d417fce448617de4a9

Scope

The obsolete unrelated MMMU skip was removed while refreshing this branch onto current main.

Note

Medium Risk
Prime CLI now defaults eval dispatch through the v1 bridge (behavior change for all Prime eval invocations), plus remote sandbox deletion and timeout semantics that affect rollout lifecycle.

Overview
Adds external v1 plugin loading for tasksets, harnesses, and judges: the loader tries built-in namespaced imports first, then top-level installed packages, surfaces real dependency errors from a found plugin, and documents the packaging ABI (__all__, ID normalization, typed config).

Introduces eval_v1_prime as the Prime CLI eval entrypoint. v1 TOMLs with [env.taskset] route to eval @ config.toml after stripping Prime-injected v0 flags; other targets still use legacy eval. Unsupported bridge args fail closed.

Prime and Modal gain configurable hard timeout (60–86400s), stricter idle_timeout validation on Prime, provider forwarding of lifetime limits, and stop_confirmed() for fail-closed teardown (404-as-gone on Prime, idempotent success, stopped=True). Harness timeouts are capped against the runtime's configured timeout instead of a fixed 24h.

Docs and tests cover Prime v1 eval usage, external harness packaging, loader regressions, compile capping, and runtime stop behavior.

^{Reviewed by Cursor Bugbot for commit 80ab00a. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add external harness support and v1 eval bridge for the Prime CLI plugin

Adds eval_v1_prime.py as the new Prime CLI plugin eval module, routing .toml files containing [env.taskset] to the v1 evaluator and falling back to the legacy v0 path; unknown flags fail closed.
Updates loaders.py to resolve external harness plugins by trying the built-in namespace first, then the flat top-level module; internal import errors surface directly instead of being swallowed.
Adds stop_confirmed to Prime and Modal runtimes with idempotency, 404-as-success handling for Prime, and cleanup guarantees.
Adds a configurable timeout field (60–86400 s) to PrimeConfig and ModalConfig; sandbox creation now uses this value instead of a fixed 24 h ceiling, and harness timeouts are capped against it.
Risk: the Prime CLI plugin now routes all v1 TOML evals through the new bridge; any unrecognized flag passed by the CLI host will cause a hard failure.

^{Macroscope summarized 80ab00a.}

macroscopeapp · 2026-07-13T18:48:49Z

Approvability

Verdict: Needs human review

Unable to check for correctness in 80ab00a. This PR introduces new external harness support with significant runtime lifecycle changes. Three unresolved medium-severity bugs were identified in the review comments affecting the new stop_confirmed() methods - including lifecycle state not being reset on restart and potential resource leaks in Modal cleanup paths.

^{You can customize Macroscope's approvability policy. Learn more.}

sethkarten · 2026-07-17T09:04:03Z

@samsja This is ready for human review at 578cae0451413f37e3635d2d5c12a6e18c3b4d61.

All required checks pass on Python 3.11 through 3.13, Ruff, Ty, Environments, CodeQL, and Cursor. The change remains provider-generic and adds bounded confirmed teardown, hard runtime lifetime propagation, provider-ID preservation, and Prime idle-timeout validation. Research-environments #678 is the downstream consumer and is also green. No sandbox or evaluation was launched.

sethkarten · 2026-07-17T09:10:40Z

Follow-up 9f1417dcc1ea937adf67b81dc3766ffa9b43bacf changes only a test fixture label from a benchmark-specific example to generic-runtime-test, keeping this PR fully provider-generic. The focused 9-test file, Ruff, format, and pre-push Ty checks pass. Production package bytes are unchanged; the full CI matrix is rerunning.

sethkarten · 2026-07-18T14:27:09Z

Friendly follow-up @samsja: this remains automated-review green at 9f1417dcc1ea937adf67b81dc3766ffa9b43bacf with zero unresolved threads. Human approval is the only remaining PR gate; the latest commit is test-only and keeps the Verifiers diff benchmark-generic.

sethkarten · 2026-07-20T23:02:21Z

@samsja Review update for current head 2b916dacaba7cb9706c13514a26e3dd2c8666a82:

the generic example is now SuperSecretHarness / SuperSecretHarnessConfig
Python 3.11 and 3.13 passed on the initial run
the unrelated stochastic Python 3.12 live-test failure passed on rerun, so the complete required CI set is green
Ruff, Ty, CodeQL, Cursor, and Macroscope correctness checks are green or neutral as expected
the PR remains provider-generic, mergeable, and has no unresolved review threads

Please review the current head when available. No sandbox, model request, or evaluation was launched.

stop_confirmed raised on 404 when a Prime sandbox was already removed by idle timeout, max lifetime, or earlier best-effort cleanup, leaving _confirmed_stop_id unset so every retry kept failing. Now a 404 is treated as confirmed success; other delete errors still propagate. Addresses Cursor Bugbot high-severity finding on PR #1985.

Resolve conflict in verifiers/v1/loaders.py by keeping the branch's robust external plugin loading (dual-target import with dependency error filtering) and merging main's default→bash rename hint and builtin_harness_ids hint. Update test_loaders_external_plugins.py for main's config API (taskset on env, harness on agent, Environment→Env).

stop_confirmed now checks isinstance(exc, APIError) with an 'HTTP 404' prefix, not a loose '404' substring, so unrelated errors can't be misread as confirmed deletion. Add regression tests for non-APIError and non-404 APIError rejection. Add the timeout field to PrimeConfig and ModalConfig in the evaluate-environments skill reference, addressing the Bugbot finding about omitted config contract fields.

…t runtime lifetime stop_confirmed in PrimeRuntime and ModalRuntime now set self.stopped after confirmed teardown, matching the base stop() path, so borrow checks reject a dead runtime and no new rollout is placed into an already-deleted sandbox. cap_remote_harness_timeout now caps at the runtime's configured timeout (via getattr fallback to 24h), not a fixed 24h, so a shorter hard lifetime correctly bounds the harness timeout. Add regression tests for both fixes.

…ider ID When PrimeRuntime.start fails before assigning info.id, the live _client remains set. stop_confirmed previously raised without closing it, leaking the client. Now the client is released before raising, matching teardown's cleanup path. Add regression test.

…etime Modal truncates the hard timeout to int() before sending it to the provider, but cap_remote_harness_timeout compared against the full float. A fractional lifetime could leave the harness limit above the sandbox's actual lifetime. Now the cap uses int() to match the provider's truncation. Add regression test.

…-v1-plugins

macroscopeapp · 2026-07-24T04:26:07Z

                        **{k: v for k, v in options.items() if v is not None},
                    )
                )
+            self._confirmed_stop_id = None


🟡 Medium runtimes/prime.py:165

start() resets self._confirmed_stop_id to None when provisioning a new sandbox but does not reset self.stopped, which stop_confirmed() sets to True. After a restart of a PrimeRuntime that was previously confirmed-stopped, a new sandbox is created and running, yet self.stopped remains True, so any lifecycle check that gates on that flag incorrectly treats the live sandbox as dead. Reset self.stopped = False next to the _confirmed_stop_id reset.

Suggested change

self._confirmed_stop_id = None

self._confirmed_stop_id = None

self.stopped = False

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/runtimes/prime.py around line 165: `start()` resets `self._confirmed_stop_id` to `None` when provisioning a new sandbox but does not reset `self.stopped`, which `stop_confirmed()` sets to `True`. After a restart of a `PrimeRuntime` that was previously confirmed-stopped, a new sandbox is created and running, yet `self.stopped` remains `True`, so any lifecycle check that gates on that flag incorrectly treats the live sandbox as dead. Reset `self.stopped = False` next to the `_confirmed_stop_id` reset.

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2e0103f. Configure here.}

cursor · 2026-07-24T04:27:56Z

                        **{k: v for k, v in options.items() if v is not None},
                    )
                )
+            self._confirmed_stop_id = None


Restart leaves runtime unborrowable

Medium Severity

start clears _confirmed_stop_id after a successful create so a runtime can be restarted after stop_confirmed, but it never resets stopped to False. A restarted box therefore stays permanently rejected by borrow checks even though a live sandbox was provisioned again.

Additional Locations (1)

verifiers/v1/runtimes/modal.py#L109-L111

^{Reviewed by Cursor Bugbot for commit 2e0103f. Configure here.}

cursor · 2026-07-24T04:27:56Z

+        if runtime_id is None:
+            raise RuntimeError(
+                "modal sandbox termination cannot be confirmed without a provider ID"
+            )


Modal skips terminate without ID

Medium Severity

When Modal still has a live _sandbox handle but info.id is missing, stop_confirmed raises without calling terminate or releasing the handle. Controllers that only use the confirmed path then cannot clean up a sandbox they already hold a handle to, so the remote resource leaks until max lifetime.

^{Reviewed by Cursor Bugbot for commit 2e0103f. Configure here.}

macroscopeapp Bot previously approved these changes Jul 13, 2026

View reviewed changes

sethkarten dismissed macroscopeapp[bot]’s stale review via 2bd29f2 July 13, 2026 19:05

fix(v1): harden external plugin loading

cf6e274

sethkarten force-pushed the feat/generic-external-v1-plugins branch from 2bd29f2 to cf6e274 Compare July 16, 2026 19:08

sethkarten changed the title ~~test(v1): cover generic external harness plugins~~ fix(v1): harden external plugin loading Jul 16, 2026

macroscopeapp Bot previously approved these changes Jul 16, 2026

View reviewed changes

feat(v1): confirm remote runtime teardown

93efad8

sethkarten dismissed macroscopeapp[bot]’s stale review via 93efad8 July 17, 2026 04:05

cursor Bot reviewed Jul 17, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/prime.py

fix(v1): bound Prime idle timeout

578cae0

sethkarten requested a review from samsja July 17, 2026 05:19

test(v1): keep runtime fixtures provider-generic

9f1417d

sethkarten changed the title ~~fix(v1): harden external plugin loading~~ External Harness Support Jul 20, 2026

fix: make confirmed runtime teardown idempotent

d6d52d7

sethkarten changed the title ~~External Harness Support~~ Support external v1 plugins and confirmed teardown Jul 20, 2026

sethkarten changed the title ~~Support external v1 plugins and confirmed teardown~~ Support external harness Jul 20, 2026

docs: rename generic external harness example

2b916da

cursor Bot reviewed Jul 20, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/prime.py

cursor Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/prime.py

Comment thread verifiers/v1/runtimes/prime.py

Comment thread verifiers/v1/runtimes/prime.py

chore: trigger CI on updated head

5231b11

cursor Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/prime.py

cursor Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/prime.py

cursor Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/prime.py

cursor Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/prime.py

cursor Bot reviewed Jul 22, 2026

View reviewed changes

Comment thread verifiers/v1/runtimes/modal.py

sethkarten added 3 commits July 22, 2026 11:11

Merge remote-tracking branch 'origin/main' into feat/generic-external…

b8677be

…-v1-plugins

Update Prime runtime test for policy API

2e0103f

macroscopeapp Bot reviewed Jul 24, 2026

View reviewed changes

cursor Bot reviewed Jul 24, 2026

View reviewed changes

sethkarten added 2 commits July 27, 2026 20:13

fix: route Prime eval configs through Verifiers v1

88856d8

style: satisfy current Ruff checks

80ab00a

	self._confirmed_stop_id = None
	self._confirmed_stop_id = None
	self.stopped = False

Uh oh!

Conversation

sethkarten commented Jul 13, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

Scope

Add external harness support and v1 eval bridge for the Prime CLI plugin

Uh oh!

macroscopeapp Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Approvability

Uh oh!

Uh oh!

sethkarten commented Jul 17, 2026

Uh oh!

sethkarten commented Jul 17, 2026

Uh oh!

sethkarten commented Jul 18, 2026

Uh oh!

Uh oh!

sethkarten commented Jul 20, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

macroscopeapp Bot Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jul 24, 2026

Choose a reason for hiding this comment

Restart leaves runtime unborrowable

Uh oh!

cursor Bot Jul 24, 2026

Choose a reason for hiding this comment

Modal skips terminate without ID

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

sethkarten commented Jul 13, 2026 •

edited by macroscopeapp Bot

Loading

macroscopeapp Bot commented Jul 13, 2026 •

edited

Loading