Skip to content

Make TS adapter spawning degrade gracefully instead of silently or fatally - #996

Open
alongd wants to merge 1 commit into
mainfrom
warn-on-zero-ts-adapters
Open

Make TS adapter spawning degrade gracefully instead of silently or fatally#996
alongd wants to merge 1 commit into
mainfrom
warn-on-zero-ts-adapters

Conversation

@alongd

@alongd alongd commented Aug 18, 2026

Copy link
Copy Markdown
Member

Two failure modes in Scheduler.spawn_ts_jobs, both found on the same campaign run, both costing work that had already succeeded.

1. No eligible adapter — silently

An adapter is spawned only if it is in both the configured ts_adapters and ts_adapters_by_rmg_family[rxn.family]. When those lists do not intersect, the loop runs to completion, spawns nothing, and logs nothing.

Nothing recovers from it:

  • tsg_spawned is latched True before the loop, so the reaction is never revisited;
  • admit_unknown_family requires not family_known, so a known family with no eligible adapter cannot fall back to ts_adapters_for_unknown_unimolecular;
  • the TS guess report is written with successful_methods: [] and unsuccessful_methods: [] — both empty, because nothing was attempted;
  • the only user-visible symptom is TS did not converge, hours later, indistinguishable from having tried every adapter and failed.

Hit when ~/.arc/settings.py pinned:

ts_adapters = ['heuristics', 'AutoTST', 'GCN', 'xtb_gsm', 'orca_neb']

predating linear joining ARC's own default in arc/settings/settings.py:114. The family 1,2_Insertion_CO admits only ['kinbot', 'goflow', 'rits', 'linear']:

EFFECTIVE ts_adapters (after ~/.arc override): ['heuristics', 'autotst', 'gcn', 'xtb_gsm', 'orca_neb']
eligible for 1,2_Insertion_CO:                 ['kinbot', 'goflow', 'rits', 'linear']
--> adapters that spawn: []   (count=0)

reproducing the observed report exactly:

TS0:
  family: 1,2_Insertion_CO
  successful_methods: []
  unsuccessful_methods: []
  ts_guesses: {}

Worth naming the general hazard: arc/imports.py does settings.update(local_settings_dict), so the home settings file overrides repo defaults key-by-key. A ts_adapters list written once and left alone silently loses access to every family whose adapters were added later. It fails closed, quietly, per-family.

2. An uninstalled optional adapter — fatally

Several TS adapters shell out to a separate conda env and repo checkout. KinBot (kinbot_ts.py:253) and AutoTST (autotst_ts.py:216) signal "backend not installed" by raising FileNotFoundError out of execute_incore. That propagated through run_jobspawn_ts_jobsspawn_post_opt_jobs and terminated the entire ARC run.

The observed cost, in consecutive log lines:

Linear successfully found 7 TS guesses for s0_CH2O2 <=> s1_H2O + s2_CO.
...
ERROR: ARC crashed with <class 'FileNotFoundError'>:
The KinBot python executable was not found. Make sure the kinbot_env exists and
KINBOT_PYTHON is configured...

All 7 guesses, and every converged species job before them, lost to a missing optional dependency — one the user opted into precisely because a multi-adapter search is supposed to be redundant.

The fix

Warn in case 1, with the three facts needed to act: what was configured, what the family admits, and that no guess will be produced. Record-and-continue in case 2, appending to unsuccessful_methods.

Deliberate choices:

  • Not raising in case 1 — a user may legitimately skip TS work for some families, and the loop also runs for reactions whose TS was supplied by hand.
  • Not auto-adding an adapter — silently changing the user's method set is how case 1 became invisible in the first place.
  • FileNotFoundError only, not a bare except — it is the documented "backend missing" signal for both adapters that have one, and a broader catch would mask real defects.

Verification

$ python -m pytest arc/scheduler_test.py -q -k spawn_ts_jobs
3 passed in 6.26s

$ python -m pytest arc/scheduler_test.py -q
11 passed, 35 errors        # baseline on main: 9 passed, 35 errors

The 35 errors are pre-existing on main and unrelated — KeyError: 'server1', the test fixture's template servers dict shadowed by a real ~/.arc/settings.py. Same override mechanism as case 1 above, and worth fixing separately.

@alongd
alongd force-pushed the warn-on-zero-ts-adapters branch from 72010d0 to 071d6e2 Compare August 18, 2026 07:15
@alongd alongd changed the title Warn when no TS adapter is eligible for a reaction instead of failing silently Make TS adapter spawning degrade gracefully instead of silently or fatally Aug 18, 2026
@codecov

codecov Bot commented Aug 18, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.36%. Comparing base (d033cbb) to head (21ae360).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #996      +/-   ##
==========================================
- Coverage   64.38%   64.36%   -0.03%     
==========================================
  Files         119      119              
  Lines       39601    39609       +8     
  Branches    10269    10271       +2     
==========================================
- Hits        25499    25495       -4     
- Misses      11121    11130       +9     
- Partials     2981     2984       +3     
Flag Coverage Δ
functionaltests 64.36% <ø> (-0.03%) ⬇️
unittests 64.36% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the robustness and debuggability of TS adapter spawning in Scheduler.spawn_ts_jobs() by (1) avoiding whole-run termination when an optional TS backend is missing, and (2) emitting an explicit warning when no TS adapter is eligible for a reaction (instead of failing silently).

Changes:

  • Catch FileNotFoundError from TS search job spawning, record the adapter in unsuccessful_methods, and continue to the next adapter.
  • Warn when the configured ts_adapters list has no eligible adapter for a known/unknown RMG family, making the “no jobs spawned” condition visible.
  • Add regression-style unit tests documenting the two failure modes.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
arc/scheduler.py Adds graceful handling for missing optional TS adapter backends and warns when no TS adapter is eligible to spawn.
arc/scheduler_test.py Adds regression tests intended to cover “no eligible adapter” and “missing backend” scenarios.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread arc/scheduler.py Outdated
Comment thread arc/scheduler_test.py Outdated
…f silently or fatally

Two failure modes in ``Scheduler.spawn_ts_jobs``, both found on the same
campaign run, both costing work that had already succeeded.

1. No eligible adapter, silently.

An adapter is spawned only if it is in BOTH the configured ``ts_adapters`` and
``ts_adapters_by_rmg_family[rxn.family]``. When those lists do not intersect the
loop completes without spawning anything and says nothing. Nothing recovers:
``tsg_spawned`` is latched True before the loop, so the reaction is never
revisited, and the TS guess report is written with ``successful_methods: []``
AND ``unsuccessful_methods: []`` -- both empty, because nothing was attempted.
The only user-visible symptom is "TS did not converge", much later, which is
indistinguishable from having tried every adapter and failed.

Hit when ~/.arc/settings.py pinned

    ts_adapters = ['heuristics', 'AutoTST', 'GCN', 'xtb_gsm', 'orca_neb']

predating 'linear' joining ARC's own default in arc/settings/settings.py. The
family 1,2_Insertion_CO admits only ['kinbot', 'goflow', 'rits', 'linear'].
Empty intersection, zero jobs, no log line. Because arc/imports.py has the home
settings override repo defaults key-by-key, a stale local file disables whole
families this way.

2. An uninstalled optional adapter, fatally.

Several TS adapters shell out to a separate conda env and repo checkout. KinBot
and AutoTST signal "backend not installed" by raising FileNotFoundError out of
``execute_incore``. That propagated through run_job -> spawn_ts_jobs ->
spawn_post_opt_jobs and terminated the entire ARC run.

Observed cost: 'linear' reported

    Linear successfully found 7 TS guesses for s0_CH2O2 <=> s1_H2O + s2_CO.

and then the run died on the next adapter in the list because KINBOT_PYTHON was
unset. All 7 guesses, and every converged species job before them, were lost to
a missing optional dependency.

Record the unavailable adapter in ``unsuccessful_methods`` and continue -- the
remaining adapters are exactly the redundancy that makes a multi-adapter TS
search worth configuring. The catch is narrowed to FileNotFoundError so it
cannot mask real defects.
@alongd
alongd force-pushed the warn-on-zero-ts-adapters branch from 071d6e2 to 21ae360 Compare August 19, 2026 20:51
@alongd
alongd requested a review from kfir4444 August 19, 2026 21:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants