Skip to content

fix(slm-backend): retain the last five discarded background-task launches (#15524) - #16880

Closed
mrveiss wants to merge 6 commits into
mainfrom
issue-15524-create-task
Closed

mrveiss wants to merge 6 commits into
mainfrom
issue-15524-create-task

Conversation

@mrveiss

@mrveiss mrveiss commented Sep 17, 2026 •

Copy link
Copy Markdown
Owner

Thinking Path

asyncio.create_task() is held by the event loop only by weak reference, so a call whose
return value is discarded can be collected before its coroutine ever runs. #15522 diagnosed this
live: one firing executed end to end, an identical firing minutes later produced no executor call,
no inventory file and no log write — and the surface reported success both times, because nothing
awaits these tasks, so a raise inside the coroutine has nowhere to surface either.

#15524 enumerated 12 remaining sites. Measuring against merged main first rather than trusting
the issue's table, 7 were already gone: api/stateful.py (3) went through fire_and_forget,
services/blue_green.py (1) took a retained assignment, and monitoring/performance_dashboard.py
(1) and services/deployment.py (2) no longer contain the call at all — those two left by file
removal/restructure rather than by conversion, which is worth stating plainly rather than ticking
as converted work.

The census constant has also drifted from the issue's text: #15619 and #15637 renamed
KNOWN_DISCARDED_CREATE_TASK to KNOWN_DISCARDED_LAUNCHES and widened it from one root to five.
The autobot-backend/ population it picked up is tracked separately by #15635, so #15524's
{} criterion means the autobot-slm-backend/ rows — which this PR empties.

What Changed

Five discarded launches converted to autobot_shared.async_compat.fire_and_forget, which holds a
hard reference until completion and logs a failure that otherwise vanishes:

File Line Launch
autobot-slm-backend/api/infrastructure.py 494 playbook execution
autobot-slm-backend/api/setup_wizard.py 1258 fleet provisioning
autobot-slm-backend/api/updates.py 645 update discovery
autobot-slm-backend/slm/agent/agent.py 493 code-change notification
autobot-slm-backend/ansible/roles/slm_agent/files/slm/agent/agent.py 493 vendored copy of the above

Each gets a name= carrying the job or execution id, so a failure names the work that was lost.

repo_tests/background_task_retention_ratchet_test.py: the five rows come out of
KNOWN_DISCARDED_LAUNCHES in the same commit, as the shrink-only contract requires. That empties
the root, and test_every_scan_root_is_either_censused_or_a_proven_zero requires every scan root
to be one or the other — so autobot-slm-backend/ moves into PROVEN_ZERO_ROOTS, where
test_a_proven_zero_root_really_is_zero re-measures the claim on the tree every run instead of
taking the entry's word. The module docstring is corrected to match.

Verification

  • The root really is empty — swept every create_task / ensure_future /
    run_coroutine_threadsafe call in autobot-slm-backend/ outside tests: 20 calls, all assigned
    or returned, 0 discarded. Checked independently of the AST detector, and the two agree.
  • No false retention among those 20 — a name assigned and never referenced again is a local
    that dies at return, which the detector's bare-expression rule cannot see. Every one of the 11
    plausible cases resolves to a real holder; main.py:283 is the tightest and stores into
    _code_sync_mod._running_tasks[_job_id] on the next line.
  • Both vendored copies byte-identical (diff -q, clean) — AC3.
  • asyncio still used in all four edited files (3/7/5/2 references), so no import is orphaned.
  • All six changed files parse (ast.parse).
  • The population floors and FIX THE SWEEP assertions (AC4) are CI's to prove; they are not
    claimed here.

Not run locally: this repository's code is never executed from the checkout — CI is the evidence.

Model Used

Claude Opus 5

Closes #15524

Single-issue rationale

#15524 cannot ride with another issue because its ratchet contract forbids it. The census in
repo_tests/background_task_retention_ratchet_test.py is exact and shrink-only: every conversion
must lower the census in the same commit, and emptying the root additionally moves it into
PROVEN_ZERO_ROOTS. A second issue's changes in the same PR would make a failure of that guard
ambiguous between the two.

The obvious batching partner is #15635, which owns the remaining autobot-backend/ census. It is
explicitly not ridden with: it is blocked on decomposing 19 grandfathered files, a different and far
larger piece of work at a different risk level.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Improved the reliability of background operations, including fleet provisioning, discovery, infrastructure playbook execution and code-change notifications.
    • Background tasks are now scheduled more consistently and have clearer identification, while existing request, response and notification behaviour remains unchanged.
  • Tests

    • Updated background-task tracking to confirm that previously identified discarded launches in the SLM backend have been addressed.

…ches (#15524)

`asyncio.create_task()` is held by the loop only by weak reference, so a call
whose return value is discarded can be collected before its coroutine runs —
the defect diagnosed live in #15522, where one firing executed end to end and
an identical firing minutes later produced no executor call, no inventory file
and no log write, while the surface reported success both times.

#15522 converted the three `_ansible_self_update` firings. This converts the
five that remained in `autobot-slm-backend/`:

  api/infrastructure.py:494   playbook execution
  api/setup_wizard.py:1258    fleet provisioning
  api/updates.py:645          update discovery
  slm/agent/agent.py:493      code-change notification (both vendored copies)

Each goes through `autobot_shared.async_compat.fire_and_forget`, which holds a
hard reference until completion and logs a failure that otherwise has nowhere
to surface — nothing awaits these tasks.

The census in `repo_tests/background_task_retention_ratchet_test.py` is exact
and shrink-only, so the five rows come out in the same commit. That empties the
root, and the guard requires every scan root to be either censused or a proven
zero, so `autobot-slm-backend/` moves into `PROVEN_ZERO_ROOTS` — where the zero
is re-measured on the tree every run rather than recorded as finished.

The two vendored copies of `slm/agent/agent.py` stay byte-identical.

Closes #15524
@coderabbitai

coderabbitai Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Review Change StackReview Change Stack

Important

Review skipped

Review was skipped due to path filters

⛔ Files ignored due to path filters (2)
  • repo_tests/python_file_size_ratchet_baseline.py is excluded by !repo_tests/python_file_size_ratchet_baseline.py
  • scripts/python_file_size_known_large.py is excluded by !scripts/python_file_size_known_large.py

CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including **/dist/** will override the default block on the dist directory, by removing the pattern from both the lists.

⚙️ Run configuration

Configuration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 5759763c-3956-49c3-8ade-85337822c0bc

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 94569728-4731-4251-8159-c0b3c4ebef39

📥 Commits

Reviewing files that changed from the base of the PR and between 538c332 and 888006f.

⛔ Files ignored due to path filters (2)
  • repo_tests/python_file_size_ratchet_baseline.py is excluded by !repo_tests/python_file_size_ratchet_baseline.py
  • scripts/python_file_size_known_large.py is excluded by !scripts/python_file_size_known_large.py
📒 Files selected for processing (5)
  • autobot-slm-backend/ansible/roles/slm_agent/files/slm/agent/agent.py
  • autobot-slm-backend/api/infrastructure.py
  • autobot-slm-backend/api/setup_wizard.py
  • autobot-slm-backend/api/updates.py
  • autobot-slm-backend/slm/agent/agent.py

Included review availability: Your plan provides up to 10 included reviews per hour; 2 remain after this review.


📝 Walkthrough

Walkthrough

Five discarded asyncio.create_task calls in the SLM backend now use named fire_and_forget calls. Discovery job timestamps use time_utils. The retention ratchet records the conversions and tracks autobot-slm-backend/ as a proven-zero root.

Changes

Background task retention

Layer / File(s) Summary
Convert discarded task launches
autobot-slm-backend/.../agent.py, autobot-slm-backend/api/*.py, autobot-slm-backend/slm/agent/agent.py
Five background task launches now use fire_and_forget with descriptive task names. Discovery timestamp calls use time_utils.utc_timestamp(). Existing task arguments and notification behaviour remain unchanged.
Update retention census
repo_tests/background_task_retention_ratchet_test.py
The census removes the converted discarded launches and adds autobot-slm-backend/ to PROVEN_ZERO_ROOTS.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Bug fix · Severity of issue fixed: Medium

Merge Risk: ⚪ Minimal · up to 88800

The background work remains scheduled and retained, with failures logged and the retention census still enforcing coverage. No merge-blocking risk is established.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly identifies the main change: fixing five discarded background-task launches in slm-backend. It is specific and concise enough for the changeset.
Linked Issues check ✅ Passed Issue #15524 requirements are met. The five remaining discarded launches in autobot-slm-backend/ now use autobot_shared.async_compat.fire_and_forget. The helper retains tasks and logs failures. Ea…
Out of Scope Changes check ✅ Passed The changed application files directly implement Issue #15524. The ratchet update verifies the reduced discard census and preserves scan coverage. The import and timestamp changes support the converte…
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 6 files.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Notice: 29 open PRs — past the runaway threshold (25)

There is no PR queue limit, and this is not a request to defer this PR. Work proceeds one issue at a time without a cap on open PRs; review capacity is the constraint.

This notice only means the count is high enough to be worth a glance for a runaway — something opening PRs in a loop, or a merge pipeline that has stalled so nothing is draining.

Currently open:

If the queue is draining normally, ignore this. Otherwise:

  1. Check whether CI is dispatching at all — see the ci-dispatch-watchdog status on these PRs
  2. Merge the ones whose CI has finished and review has passed: gh pr merge <number> --squash --delete-branch
  3. Look for a loop opening near-identical PRs

Warn-only runaway detector — .github/workflows/pr-queue-gate.yml. It never blocks a merge.

@github-actions

Copy link
Copy Markdown
Contributor

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.

isort failed `code-quality` on autobot-slm-backend/api/setup_wizard.py: the
fire_and_forget import was placed above `from api.websocket`, but the file's
first-party block sorts api < autobot_shared < config.

The other three converted files were already right — infrastructure.py has
autobot_shared before services, updates.py puts async_compat before time_utils,
and agent.py keeps the plain `from autobot_shared import ...` ahead of the
submodule import. Checked rather than assumed, since the same edit pattern was
applied to all four.
…ilings (#15524)

`code-quality` failed: all five converted files sat exactly 4 lines over their
recorded ceilings. Grandfathered files may not grow (#14236) — the exemption
freezes the size it was granted for — so the change had to fit rather than the
ceilings move.

The 4 was structural, not incidental: one import line, plus a 1-line
`asyncio.create_task(...)` becoming a 4-line `fire_and_forget(...)`. Each file
is now net zero, and each by a different route because the files differ:

  agent.py (both copies)  the existing `from autobot_shared import env_utils,
                          time_utils` absorbs `async_compat` on the same line,
                          so the import costs nothing; the call is one line.
  updates.py              same trick, reached by switching
                          `from autobot_shared.time_utils import utc_timestamp`
                          to the module form and qualifying its four call sites.
  infrastructure.py       the `# Start execution in background` comment above the
                          call is what `fire_and_forget` now says in its own
                          name. Removing it pays for the import.
  setup_wizard.py         a four-line `logger.info(...)` that fits on one at 82
                          characters — its magic trailing comma was what kept
                          Black from joining it.

setup_wizard.py lands 2 UNDER, so its ceiling drops 1354 -> 1352 in both twin
registries. A ratchet that can move down should.

What the compaction cost, stated rather than buried: `name=` is now the bare
identifier (`execution_id`, `job_id`, `commit[:12]`) instead of a prefixed
label like `infrastructure-playbook:{execution_id}`. The prefixed form pushed
the single line past 120 characters. The identifier alone is still unique and
traceable in the failure log, which is what the name is for; the prefix was
readability, and it was not worth a ceiling raise.

Both vendored copies of agent.py stay byte-identical (`diff -q`, clean), and no
line in any of the five exceeds 120 characters.
@mrveiss

mrveiss commented Sep 19, 2026

Copy link
Copy Markdown
Owner Author

Carried by vehicle #17086, which includes this PR's approved head e057b4efa. Closed now as carried, per the owner's ruling (2026-09-19) that consolidated work shouldn't keep open duplicates or trigger extra CI. The branch is kept. The vehicle's own Closes lines close the linked issues when it lands. If #17086 is abandoned, this PR gets reopened.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend bug Something isn't working priority: high

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Discarded asyncio.create_task calls can be garbage-collected mid-flight (12 sites)

1 participant