Repository navigation
[Feat] Add bounded recovery for stalled browser and desktop tasks - #31
Conversation
prettygirlisnotme
left a comment
prettygirlisnotme
left a comment
There was a problem hiding this comment.
Author self-review of 72bf502 (current head):
- Budget and execution: attempts are charged before the read-only refresh; active refresh/planning time accumulates across the task and progress does not refund it. Plans guide the next ordinary decision rather than executing tools. Browser recovery remains opt-in, the existing tool filtering/permission path remains in force, and the unbounded game branch is preserved.
- Failure semantics: failed observations, empty planner responses, cancellation and exhausted recovery budgets retain their recorded reason. Post-replan BLOCKED remains a failure even with a partial answer. The targeted regressions cover empty/whitespace/null plans, retained observations and attempts, and no renewed browser decision after stopping.
- Existing validation at this head: 640 tests passed, 41 skipped, 106 subtests passed; formatting, lint, type checks, CLI smoke and offline packaging passed. These are the recorded pre-publication results, not a claim that upstream CI has passed.
- Evidence limits: the report and trajectories separate scripted mechanism checks from the 24 real-model fixture trials. Original and exploratory matrices are reported separately. The small sample does not establish general task-success gains, and the final reporting/blank-plan fixes were regression-tested after the measured revision.
The remaining integration check is upstream CI: the latest run is action_required with no jobs executed. Could a maintainer approve the workflow run? The upstream OS/Python matrix is still unverified.
This is an author self-review to make the change easier to assess, not an independent approval.
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
One P2 finding in the custom-task evaluation path.
| logs_dir = logs_root / f"{task.name}__{arm}__r{index}" | ||
| started = time.perf_counter() | ||
| try: | ||
| episode, record = await run_trial(task, arm, index, config=config, logs_dir=logs_dir) |
There was a problem hiding this comment.
[P2] Forward custom tasks to the per-trial fixture
run_eval(tasks=...) iterates the supplied tasks, but this call omits tasks=tasks, so run_trial starts its fixture with DEFAULT_TASKS. A task with a new path gets a 404 in both arms; one reusing a default path can silently run the old fixture mechanics instead of the requested variant. Please forward tasks=tasks here and add a regression that checks the fixture served for a custom task.
There was a problem hiding this comment.
Fixed in bd54c14. Each trial now receives the requested task set. The regression covers a new route and a reused default route with different page behavior and submission validation, using the fixture's HTTP server in both arms.
Local checks passed: 641 tests, formatting, lint, types, smoke, and packaging.
|
@prettygirlisnotme Please add a system1-omni inference trial and a short video of the agent's recovery workflow. The real-model report uses in-process Laya 0.3.5 on four CPU threads. System1-Omni supports served Laya on CPU/MPS and native CUDA; #35 supplies the agent-side Please attach a video showing a stall, refreshed observation, recovery plan, next model-selected action and independently checked outcome. Include a representative failure if available, exact agents/engine commits and command, and link the trace. Preserve the distinction between scripted mechanics checks and real-model runs. |
|
I ran the System1-Omni Python CPU worker through the I checked the loaded worker windows: 512/192 for the checkpoint defaults, plus a private 4096/1536 override, with matching client thresholds. Both used 4 CPU threads. The worker uses Laya 0.3.20 versus 0.3.5 in the earlier in-process run, so this is not a pure transport comparison. All eight planned trials are retained. Success, 1x, no temporal cuts: stall → fresh observation → real plan → Laya types and submits → independent check. Trace. success.mp4Failure, 3x, no temporal cuts: two correct unlock-first plans, followed by Enter actions and failure.mp4Reproduction archive, including uncropped recordings. Earlier scripted mechanics evidence remains separate. Recordings: @prettygirlisnotme, project fixtures. Reuse in project updates requires my approval. |
|
@prettygirlisnotme We're prioritizing the new backends and agent improvements for v0.2.0. Thanks for the System1-Omni served-Laya follow-up, retained failures, success/failure recordings and custom-task regression fix. Please resolve the current merge conflicts, update the PR description to point to that latest evidence and its limitations, and rerun the required checks on the integrated head. Please reply with any remaining blockers and expected completion time; link any CI run that needs maintainer approval. The existing recordings can answer the earlier demo request without presenting the small fixture sample as a general success-rate improvement. |
|
Merged main Local checks passed: The full self-review found no blocking defect. I've updated the description with the review order, code/diff counts, recordings and sample limitations. The one-off reports and trial JSON remain available in the archive and fixed commits; the older evidence comment's links are fixed too. All eight trials are retained. Two small fixtures with one repeat per arm do not establish a general success-rate improvement; the experiment pins and results are unchanged. The code and local checks are complete. The upstream CI run for |
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved recovery correctness, evaluation accounting, and fixture-rendering issues prevent safe approval.
Review effort: Balanced
Findings: 1
Open (7)
What changed in this PR
Adds bounded replanning to the existing browser and desktop agent loops for #16, while keeping broader evaluations and desktop action extensions separate.
Changes:
- Shares cumulative recovery attempt and time limits across both fronts.
- Adds recovery escalation and incomplete-usage accounting.
- Adds on/off evaluation fixtures, regression tests, and documentation.
| File | Description |
|---|---|
| tests/test_tool_models.py | Tests bounded terminal outcomes. |
| tests/test_tool_loop.py | Tests recovery wiring and accounting. |
| tests/test_spec.py | Tests unknown-cost preservation. |
| tests/test_rethink_rail.py | Tests bounded replanning behavior. |
| tests/test_recovery.py | Tests shared recovery limits. |
| tests/test_recovery_metrics.py | Tests reporting calculations. |
| tests/test_recovery_live.py | Tests live evaluation helpers. |
| tests/test_recovery_eval.py | Tests browser fixtures and summaries. |
| tests/test_jobs.py | Tests usage metadata persistence. |
| tests/test_desktop_recovery_eval.py | Tests desktop evaluation helpers. |
| tests/test_desktop_env.py | Tests read-only window refresh. |
| tests/test_counting_model.py | Tests failed and interrupted calls. |
| tests/test_browser_policy.py | Tests observation failure handling. |
| tests/test_browse.py | Tests usage completeness reporting. |
| tests/test_agents_desktop.py | Tests desktop recovery integration. |
| tests/system/test_recovery_desktop.py | Adds opt-in native desktop checks. |
| tests/system/test_recovery_browser.py | Adds opt-in browser checks. |
| tests/fixtures/desktop_recovery/S1AFixture.cs | Adds independently verified desktop fixture. |
| s1a/tool/series.py | Wires limits and preserves unknown costs. |
| s1a/tool/rethink.py | Adds bounded refresh and replanning. |
| s1a/tool/models.py | Adds bounded failure terminals. |
| s1a/tool/loop.py | Integrates recovery and call accounting. |
| s1a/recovery.py | Defines shared budgets and escalation. |
| s1a/jobs.py | Persists usage completeness. |
| s1a/desktop/env.py | Adds refresh and observation-based progress. |
| s1a/counting_model.py | Records failed and cancelled calls. |
| s1a/browser/browse.py | Adds recovery flags and result handling. |
| s1a/agents/desktop.py | Configures desktop recovery defaults. |
| evals/recovery/summary.py | Adds paired evaluation summaries. |
| evals/recovery/scripted.py | Adds scripted evaluation models. |
| evals/recovery/runner.py | Runs isolated browser trials. |
| evals/recovery/fixture.py | Adds forms and submission verification. |
| evals/recovery/__main__.py | Adds evaluation CLI. |
| evals/recovery/__init__.py | Describes evaluation scope. |
| evals/README.md | Documents evaluation workflows and evidence. |
| docs/browser-front.md | Documents browser recovery behavior. |
| docs/agents.md | Documents flags and supported models. |
| CHANGELOG.md | Records recovery features and fixes. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| f"<title>Submitted {task_name}</title></head><body>\n" | ||
| f"<h1>Submitted</h1><p id='value'>value={value}</p>\n" |
| record["status"] = "ok" | ||
| record["input_tokens"] = int(reply.usage.input_tokens) | ||
| record["output_tokens"] = int(reply.usage.output_tokens) | ||
| record["usage_known"] = True |
| verified = fixture.verified(task) | ||
| status = answer.get("status") |
| status = answer.get("status") | ||
| if verified: | ||
| terminal_name = "verified" | ||
| error = None |
| """ | ||
| _validate_run(repeat, tasks, arms) | ||
| config = config or EvalConfig() | ||
| run_id = f"{datetime.now():%Y-%m-%d__%H-%M-%S}" |
| # episode right after a plan (no failed recovery event carries one), synthesise it from the shared | ||
| # helper and name the cap as the reason: never dress an act cap up as a recovery timeout. | ||
| last = next((e for e in reversed(state.rethinks) if e.get("attempt") is not None), None) | ||
| next_action = recovery_next_action(termination=reason) |
| window = self._digests[-self._stall_after :] | ||
| stalled = self._stall_after > 0 and self._since_progress >= self._stall_after and window.count(window[-1]) > 1 |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Three reproduced P2 findings are attached inline.
For cleanup, the merge-base diff contains 7,532 changed lines of authored code: 1,036 in the runtime, 2,911 in evaluators, and 3,585 in tests and the executable fixture. I recommend consolidating the three paired-summary/rendering implementations, standardizing call-count conventions, and moving optional desktop resume/WSL retry machinery into a follow-up. Keep the independent completion fixtures and behavioral tests.
Validation at 86e5f2b24c99e072aa433904565d0191fbedeada: targeted tests passed (351 passed, 2 skipped, 45 subtests passed); formatting, lint, types and diff checks passed. The findings were reproduced offline through the existing test doubles, including the production run_episode path for the truncated escalation. Real browser/desktop integration and GitHub's OS/Python matrix remain unverified.
| progressed = score != self._last_score | ||
| self._last_score = score | ||
| self._keys.append(key) | ||
| self._scores.append(score) | ||
| self._digests.append(progress_digest(result.get("state"))) | ||
| if progressed: | ||
| self._since_progress = 0 # progress clears the window; it never returns attempts or seconds |
There was a problem hiding this comment.
[P2] Treat visible desktop changes as progress
Desktop score stays zero until completion, so score-only progress tracking leaves the stall counter running while the display changes. With stall_after=3, the displays 1, 12, 12, 123, 1234, 1234, 12345, 123456, 123456 trigger recovery after each isolated no-op. A two-attempt budget gives up at action 9 despite visible progress between those no-ops. Reset the detection window on meaningful observation changes while preserving the cumulative recovery budget.
| except Exception as exc: # noqa: BLE001 - any planner failure is a clean terminal, never a retry loop | ||
| self._stop_recovery( | ||
| run, event, stage="planner", termination="error", error=f"recovery planner failed: {exc}" |
There was a problem hiding this comment.
[P2] Sanitize planner errors before persisting them
The planner exception text is copied into the terminal summary and saved recovery events. The OpenAI-compatible client wraps the provider exception text in its MODEL_CALL_FAILED error, so response bodies and echoed credentials can reach this path. A reproduction using a synthetic authentication error preserved its API key text in both the terminal and report()["recovery"]["events"]. Record a bounded, sanitized reason or exception type, consistent with CountingModel, rather than the raw provider error.
| # next_action first keeps the one actionable step inside the terminal's short output budget. | ||
| summary["next_action"] = next_action | ||
| summary["recovery"] = recovery |
There was a problem hiding this comment.
[P2] Preserve escalation as structured data
next_action follows an unbounded reason, while run_episode persists only the first 300 output characters. A reproduced long planner failure through the production episode loop loses next_action completely from episode.extra["output"], despite the full terminal containing it. Persist the terminal fields separately as structured data so the suggested next step survives; this also removes the need for the desktop evaluator's regex parsing of truncated JSON.
|
Fixed the three P2s and the evaluation issues:
The three reports now share pairing, denominators and table rendering. All use Local checks passed on the changes over Native desktop and GitHub's OS/Python matrix remain unverified on this version. The existing real-model results and recordings keep their original commit pins. The CI run for The fixes and local checks are complete; I can address any failures once upstream CI runs. |
hsliuustc0106
left a comment
hsliuustc0106
left a comment
There was a problem hiding this comment.
Summary
PR #31 adds bounded recovery for stalled browser and desktop tasks (#16). A stall spends one attempt from a shared RecoveryLimits budget (default 3 attempts / 15 active seconds), refreshes the observation read-only, and asks the chat model for a plan that only rides into the next ordinary decision — it never executes tools or widens permissions. Exhausted/failed/empty-plan recovery ends the run with a reason and an operator next action. Browser recovery is opt-in (--rethink on); desktop stalls after 3 actions. Evaluation fixtures with independent completion oracles and paired reports are included.
The change request (hsliuustc0106, 2026-10-07 14:52 UTC, on 86e5f2b, three reproduced inline P2s) is answered by commit 82d2a94 ("[Fix] Correct recovery progress and evaluation records", pushed 16:35 UTC). Verified in the worktree at that head.
Was the change request addressed?
- "[P2] Treat visible desktop changes as progress" — Addressed.
_bounded_afternow counts a changed observation digest as progress while the score stays zero,_stalled()detects both a no-op window and an A-B-A-B cycle, and_restart_window()resets only detection, never the budget (s1a/tool/rethink.py:208-222, 224-261, 370-373).WindowEnv.observenow reports the window itself (title/elements/values) asprogress(s1a/desktop/env.py:88-96). Pinned bytest_a_moving_desktop_display_is_progress_even_with_a_frozen_score,test_a_changed_display_needs_three_subsequent_noops_to_stall,test_a_two_state_cycle_still_recovers,test_attempts_never_reset_when_the_episode_makes_progress(tests/test_rethink_rail.py:376-452). - "[P2] Sanitize planner errors before persisting them" — Addressed. New
recovery_failure_reasonpersists only the exception type (or "permission denied") and never the provider message (s1a/recovery.py:30-38); used in both browser refresh/planner handlers (s1a/browser/decision_model.py:453-458, 496-501) and the tool rail's handlers (s1a/tool/rethink.py:345-363). Tests assert the synthetic key text is absent from events andepisode.extra(tests/test_rethink_rail.py:454, tests/test_tool_loop.py). - "[P2] Preserve escalation as structured data" — Addressed. The full stop summary is kept on
EvalState.terminal(s1a/tool/models.py:51, 224) and persisted whole asepisode.extra["terminal"]beside the legacy 300-charoutput(s1a/tool/loop.py:338-339), sonext_actionsurvives truncation; regression-tested with a long planner failure (tests/test_tool_loop.py). - Review-body cleanup recommendations — Addressed. All three evaluators now import pairing/denominators/rendering from
evals/recovery/summary.py; call counts standardized (model_calls = decision_calls + chat_calls, planner as subset — evals/recovery/runner.py:82-95); desktop--resume/WSL launch-retry machinery removed (evals/desktop_recovery.py). Independent fixtures and behavioral tests kept as requested.
Bonus: 82d2a94 also fixes all seven Copilot findings (fixture HTML escaping, Usage.known distinction, timeout/run-error preservation after verified submissions, run-id random suffix, permission-denied error pass-through).
Findings
1. Recovery bounds are sound — no unbounded loop. [P2, verified] Attempts are charged in begin_attempt before any await and are monotonic; asyncio.wait_for enforces the remaining seconds; RecoveryExhausted yields a give_up/BLOCKED terminal on both fronts (s1a/recovery.py:110-132; s1a/browser/decision_model.py:439-443; tests/test_recovery.py:71). The unbounded game branch is untouched when refresh/limits are absent, and mismatched configuration is rejected (s1a/tool/rethink.py:143-144).
2. WAIT/settle interaction is coherent. [P2, verified] Browser recovery consumes the existing settle-deadline and wait-streak signals (wait_settle, wait_streak — decision_model.py:338-361); after an attempt only detection windows reset (_reset_detection, line 509-516); a stopped recovery forces BLOCKED on later turns, and a post-replan BLOCKED is recorded as blocked_after_recovery, not success (line 552-586).
3. recovery_failure_reason substring match can misclassify. [P3] Any exception whose message merely contains "denied" (e.g. "connection denied") maps to "permission denied" and routes the operator to the permission flow (s1a/recovery.py:36-37). Conservative-direction, cosmetic; no fix required.
4. Upstream CI still unverified. [P2, process] gh pr checks 31 reports no checks on feat/bounded-recovery; the workflow run remains action_required (fork). Not a code defect, but the OS/Python matrix is still unproven — a maintainer workflow approval is the remaining gate.
What I ran
Worktree /tmp/pr31-review at 82d2a94; diff checked against the true merge base adcc5d1 (45 files, 8,237+/122-, matching the PR body). Full suite with the known-good recipe: 811 passed, 39 skipped, 141 subtests passed; only tests/test_cli.py::TestStdoutIsForResults::test_outside_a_checkout... failed (fails on main too, pre-existing). Secret scan of the diff: only clearly fake test doubles.
Disposition
All three requested changes and the advisory cleanup items are implemented, regression-tested, documented (docs/agents.md, docs/browser-front.md, docs/decision-models.md, CHANGELOG.md) and verified locally. No committed evidence contains secrets. Approve; remaining follow-up is upstream CI approval (and real browser/desktop integration checks stay opt-in by design).
|
fix conflicts |
|
One Windows-only failure on the current head (run 37798080295,
|
|
Withdrawing my approval — please treat it as void. I approved without checking mergeability: this branch is currently CONFLICTING with main (CHANGELOG.md, docs/agents.md, s1a/agents/desktop.py, s1a/desktop/env.py, s1a/tool/models.py, s1a/tool/series.py, tests/test_desktop_env.py). My mistake. Path back to approval: (1) resolve the conflicts with current main, (2) fix the Windows-only assertion in tests/test_recovery.py:104 (diagnosed above), (3) green CI on the resolved head, then re-request review. The substance of the review — the three requested changes being addressed, the recovery bounds — still stands; it's the mergeability and the flaky test that gate now. |
|
Resolved the seven conflicts with main ( The rejected-awaitable test now fixes the budget clock before checking rejection. It still checks that the rejected work never runs or warns; the separate real-time timeout test remains. The planner-cancellation test waits for planner entry before cancelling. I also kept the recovery plan in in-process Laya's compacted browser state. Local checks passed: 925 tests, 45 skips and 189 subtests, plus formatting, lint, types, smoke, build and lock checks. The recovery-budget file also passed all 10 tests on native Windows 3.11.16. That is not the full Windows CI suite. The existing real-model recordings retain their original commit pins; they were not rerun. I'll re-request review once CI is green on this version. |
Birfy
left a comment
Birfy
left a comment
There was a problem hiding this comment.
Reviewed current head 76f6a81. One new P2 integration finding is attached inline: screenshot-only desktop progress is invisible to the bounded recovery detector.
Validation: inspected the current diff, relevant tests and prior review fixes, and reproduced this with the exact-head WindowEnv/RethinkRail methods in a standard-library harness using a fake driver and planner. The accessibility tree/title remain fixed, image content changes on every pixel action, and capture IDs change in both the moving-image and static-image controls. With one recovery attempt, both controls recover at action 3 and give up at action 6; with the default three attempts, both plan at actions 3/6/9 and give up at action 12. No native GUI, live model or full test suite was run in this review.
Current integration gates remain: GitHub reports mergeable=false/dirty, and the exact-head test workflow is action_required (https://github.com/ThinkFlowLab/system1-agents/actions/runs/37871229447). This scoped comment does not replace the OS/Python checks on the next resolved head.
| "text_pending": bool(self._text and not self._typed), | ||
| "progress": {"values": values, "presses": len(self._presses)}, | ||
| # progress is the window itself, not the click count: a click that changes nothing must look stuck | ||
| "progress": {"title": snapshot.window.title, "elements": elements, "values": values}, |
There was a problem hiding this comment.
[P2] Include screenshot-visible changes in desktop recovery progress
For the supported CUA 4B multimodal/pixel-target path with --rethink on, a canvas can change after each successful click while its window title and accessibility elements stay unchanged. This new progress object drops the click count but contains no image-derived state; images() supplies the screenshot separately, so progress_digest() cannot see that change. In a deterministic exact-head reproduction with a new valid PNG on every alternating pixel click, recovery still planned at actions 3/6/9 and gave up at 12 despite continued visual progress, exactly like a static-image control.
Please include a stable visual-content signal in progress detection when screenshots are enabled, without serializing image bytes or treating each fresh capture ID as progress. Add a combined WindowEnv/recovery regression where changed images with the same AX tree do not consume recovery attempts, while repeated captures of unchanged pixels still detect a stall.
There was a problem hiding this comment.
Desktop progress now includes a cached fingerprint of decoded RGBA pixels and image dimensions. Capture IDs and PNG metadata are excluded.
The combined WindowEnv/RethinkRail regression covers both controls: six changing frames with the same title and AX tree spend no attempts; unchanged pixels with fresh captures plan at action 3 and stop at action 6 with one attempt. The old code fails the moving-image case.
I also resolved conflicts with main (d5e0ab3), retaining OmniJev, browser image observations and final.png.
Local checks passed: 967 tests, 47 skips and 198 subtests, plus formatting, lint, types, smoke, build and lock checks. No native GUI or live-model run was added. I'll re-request review after CI is green.
There was a problem hiding this comment.
Rechecked ac6a35053bf20f687facf65d7294ac2388a895a8: this screenshot false-stall finding is addressed.
I independently exercised the exact-head WindowEnv, RethinkRail and recovery-budget implementations in an isolated harness with a fake driver/planner and real Pillow. Six changing frames with fixed title/AX state caused no recovery; unchanged pixels with fresh capture IDs and varied PNG metadata/compression planned at action 3 and stopped at action 6 with one attempt. Fingerprints also distinguish changed dimensions, normalize equivalent RGB/RGBA pixels, and stay cached across repeated observation reads.
This is a scoped verification of the original finding, not full PR approval. The current head has no GitHub workflow/check runs. I did not rerun the full suite, native GUI, browser integration or live-model evaluation; the historical evidence remains separately pinned.
The 2026-10-07/08 review passes approved two pull requests (ThinkFlowLab#31, ThinkFlowLab#33) that were CONFLICTING with main, and both approvals had to be withdrawn in public threads: an approval issued without a mergeability check is an invalid signal, and a green CI run pinned to an older commit proves nothing about the head under review. review-pr now verifies the merge gates before any approve or ready statement (the branch reports mergeable, the required checks ran green on the exact head commit, the diff comes from a freshly fetched default branch; fork workflow runs pending maintainer approval count as pending, not absent). It also reads for readability - names that say what they do, reviewable function length, the house idioms - and the old blanket ban on style comments narrows to formatter-settled formatting, so naming and length are judged against the repository's own conventions rather than personal preference. self-review asks authors for the same gates before requesting review and after every push, with the head SHA recorded in the report.
Why
Browser and desktop agents can repeat ineffective actions without recovering or explaining how to proceed. This adds the bounded recovery subset agreed in #16. The full #13 evaluation suite is outside this PR; current main's desktop actions are retained.
Behavior
A stall spends one attempt, refreshes the observation read-only and asks the existing chat model for a short plan. The next ordinary decision chooses an action through the existing tools and permission checks. Progress does not refund either budget.
Browser recovery is opt-in with
--rethink on. Desktop detects consecutive no-ops and repeated-state cycles when rethink is on; its no-op threshold is three actions. Both fronts default to three attempts and 15 active recovery seconds. Failed, cancelled, empty-plan or exhausted recovery stops with a reason and a next action. A partial answer afterBLOCKEDcannot make the run successful.Visible desktop progress resets no-op detection even while the completion score stays zero. With screenshots enabled, progress includes decoded image dimensions and a cached RGBA pixel hash, ignoring capture IDs and PNG metadata. Pillow is declared directly; it was already a transitive dependency. Tool episodes retain full terminal fields separately from shortened display output. Bounded refresh/planner failures omit provider exception bodies and preserve permission guidance. Failed/cancelled calls stay counted, missing usage stays unknown, and a verified fixture submission does not erase a later task timeout.
Scope and review order
Authored code: 8,329 changed lines; total diff: 8,562, 51 paths, against current main (
d5e0ab30d6ee9a035005f716dc3744d6d9186ef8). Counts include additions plus deletions. Executable test-fixture source counts as code; documentation, generated output, lockfiles and static data do not.The shared budget, both agent fronts, terminal/accounting interfaces and independent result checks stay together so this feature can be reviewed with its validation. The three evaluators share pairing, denominators and table rendering. Optional desktop resume and launch retries have been removed from this PR. Evaluator code decreased from 2,911 to 2,871 lines; additional regression tests mean the overall code diff is larger. The conflicts with current main are resolved in the tested candidate, retaining OmniJev, browser image observations and final screenshots. The Windows rejection test no longer depends on exact wall-clock exhaustion. One-off results remain in the archive and fixed commits below.
s1a/recovery.pytest_recovery.pytest_desktop_visual.py,test_rethink_rail.py,test_tool_models.pyandtest_tool_loop.py: pixel changes with a fixed AX tree, unchanged pixels with fresh captures, no-ops/cycles, cumulative budget, sanitized failures and full terminal fieldstest_browser_recovery.py,test_browser_policy.py,test_browse.pyandtest_decision_models_laya.py: fresh observation, tool filtering, one-turn plans, blocked outcomes and in-process Laya plan preservation; main's image observations and final screenshot tests pass; four opt-in Chromium scenarios passed previously with scripted models; not rerun after this main synctest_decision_models_types.py,test_counting_model.py,test_jobs.pyandtest_recovery_metrics.py: errors/cancellation, explicit-zero versus unknown usage, unknown cost and call-count conventionstest_recovery_eval.py,test_recovery_live.pyandtest_desktop_recovery_eval.py: custom tasks, isolated oracles, HTML escaping, unique run directories, failure denominators and structured terminal fieldstest_agents_desktop.py,test_desktop_env.pyandtest_spec.py: refresh/budget wiring, permission fallback and legacy games; native desktop was not rerunDemo / evidence
The served-Laya report,
recordings and reply, and
reproduction archive
preserve all eight planned trials, refreshed observations, real plans, next model actions, independent POST checks and failures.
Those trials used the System1-Omni Python CPU worker, Laya 0.3.20 and the Go DeepSeek v4.1 Flash planner/value provider. The checked worker windows were checkpoint defaults 512/192 and an explicitly labelled private 4096/1536 override, with matching client thresholds and four CPU threads. The archive pins agents
bd54c14plus the then-unmerged #35365a17cf, engineae86032cand checkpoint55cf4c4. These pins and results are historical; they do not describe a rerun of the current changes.In both profiles
normal/onpassed the independent check. The locked-field task failed, including a case where two correct unlock-first plans were not followed by Laya. Two synthetic tasks with one repeat per arm do not establish general success-rate gains. Worker/in-process library versions differ, so this is not a pure transport comparison.Success video: 1x, no temporal cuts.
Failure video: 3x, no temporal cuts.
Viewport crop, trace-caption alignment and the independent oracle end cards are described in the report; uncropped originals are in the archive. Recordings are by @prettygirlisnotme using project fixtures. Reuse in project updates requires my approval.
The earlier in-process report and
24 trial records
remain separate. The original matrix verified 0/6 in both arms without triggering recovery; an exploratory validation variant verified 0/6 off and 2/6 on. Scripted browser/Windows fixtures demonstrate mechanics rather than model accuracy.
Validation
Local WSL/Linux, Python 3.12.3. Tested merge candidate: published head
76f6a810599a62826fe499a833e3ceeb4990eca8plus maind5e0ab30d6ee9a035005f716dc3744d6d9186ef8, final tree63f49e6c36aafdaf4072cb1e9324913074364efd.uv run --no-sync ruff format --check .: 222 files already formatted.uv run --no-sync ruff check .anduv run --no-sync ty check: passed.uv run --no-sync pytest -q: 967 passed, 47 skipped, 198 subtests passed.bash scripts/smoke.sh,uv build --offlineanduv lock --check --offline: passed.git diff --cached --check d5e0ab30d6ee9a035005f716dc3744d6d9186ef8: passed.WindowEnv/RethinkRailregression fails on the old moving-image behavior and passes with the fix. Six changing frames with the same title/AX tree spend no attempts; unchanged pixels with fresh capture IDs and PNG metadata plan at action 3 and stop at action 6 with one attempt.tests/test_recovery.pyusing the standard-library runner against the production budget module. That file is unchanged in this update; it was not rerun and is not the full Windows suite.The earlier opt-in
tests/system/test_recovery_browser.pycheck passed four headless-Chromium trial scenarios through productionbrowseand an independent server oracle. Its decision and planner models were scripted. It used Chromium 151.0.7922.10,@playwright/mcp@0.0.78and an explicit--executable-path; it was not rerun after this main sync.Native desktop, current-version real-model runs and the new GitHub OS/Python matrix remain unverified. The existing recordings and reports keep their original pins. In-process Laya now retains a recovery plan through compaction; this does not establish model plan-following, and the served-model recordings do not exercise that compaction path. Review will be requested again after CI passes on the resolved version.
Closes #16.