Repository navigation
bug(ci): the marker-excluded suite cannot collect — fixture 'backend_url' not found, so every integration/slow/distributed/performance test runs nowhere #14930
Description
Activity
Correcting an overstatement in this issue's own summary, before it becomes the record.
I wrote that while the suite cannot collect, "every test carrying
integration/slow/distributed/performanceruns nowhere at all." That is wrong. The suite partially runs.Actual result on PR #14929's head
4d050db81:19 failed, 35 passed, 12 skipped, 19 errors in 695.40sSo 35 marker-carrying tests do execute and pass. The failure is partial, not total: 19 errors (the
fixture 'backend_url' not foundcollection errors) plus 19 failures, against 35 genuine passes.What stands unchanged:
- The
backend_urlerrors come from tests underautobot-backend, a root already present before fix(test): tests, suites and hooks that cannot report what they were written to report (#14923, #14920, #14917) #14929 touched the file — so this is pre-existing, not introduced. - The workflow does not run on base at all (checked the last three base commits: zero runs), so nothing surfaces this. It only fires when someone edits
marker-tests.yml. - The file's own header still applies: "A red run here is a DEFECT TO TRIAGE, never an accepted steady state."
What changes: the acceptance criteria should target 19 errors + 19 failures, not a total collection failure. The fourth criterion I wrote — "the run fails loudly if it collects nothing" — is still worth having as a guard, but it is not the current symptom.
Revised scope:
- The 19
backend_urlcollection errors resolve — the fixture is provided where the run reaches it, or those tests are marked so this selection does not pick them - The 19 failures are triaged: fixed, or marked with a reason
- The run reports how many marker-carrying tests executed, so a future partial collapse is visible against today's 35
- Assert PRESENCE: the run fails loudly if it collects nothing, rather than exiting 5 and reading as clean
Correcting this here rather than leaving it, per evidence-before-done — an inaccurate issue is a bad record, and "runs nowhere" would have sent someone hunting a total failure that does not exist.
- The
Triage of both populations — one defect class, two disguises
PR #14933 targets this. The 38 non-passes are not 38 problems, and the errors and
the failures do not share a single cause the way the original framing suggested.The 19 errors — one file, one missing fixture
All 19 come from
autobot-backend/services/redis_service_api_integration_test.py.
Nothing else in the run raisedfixture 'backend_url' not found. The only definition
of that fixture lives inautobot-infrastructure/shared/tests/conftest.py, which is
not on this tree's conftest chain, so every test in the module errors at setup.The 19 failures — three causes
Count Where Cause 6 api/infrastructure_integration_test.pyConnectionRefused— backend :800110 api/voice_integration_test.pyConnectionRefused— backend :8001 and TTS worker :80832 knowledge/gpu_kb_integration_test.py,knowledge/kb_optimization_test.pyRuntimeError: Failed to initialize knowledge base1 performance_benchmarks_test.pyAssertionError: Memory usage too high—assert 1644.55 < 500So 16 of the 19 failures are the same shape as the 19 errors: real HTTP to a service
that does not exist on a GitHub-hosted runner, reported as a hard failure rather than
as a stated non-result. 35 of 38 in total.The two knowledge-base failures are NOT an infrastructure gap
This is the part worth reading. The workflow provides a healthy Redis specifically for
these, and its header says so. They fail anyway, because
autobot-backend/conftest.pyreplacesautobot_shared.redis_clientwith a
socket-free stand-in whoseget_redis_client()returnsNonefor every item
underautobot-backend/— gated on directory membership, never on markers. The
integrationtests that exist because they need real services get the no-Redis
stand-in along with everything else.The suite provisions, health-checks and configures a Redis that no backend test can
reach. Filed as #14932 with the root cause and a proposed fix, and deliberately
not fixed in #14933: it flips Redis behaviour for every backendintegrationtest
at once and 35 of them currently pass, so landing it together would make the resulting
count unattributable.The one genuine assertion defect
performance_benchmarks_test.pyasserted absolute process RSS under 500 MB. A
pytest-xdist worker sits at ~1.6 GB from importing the orchestrator, knowledge base,
memory manager and LLM service before the first benchmark runs. The budget was spent
before the test started — it could not pass, and it could not have caught a regression
in the code it names. Now measured as the RSS delta across the operation, which is
what the assertion always meant.On the choice this issue asks for
either the fixture is provided in a conftest the run reaches, or the tests requiring
a live backend are marked so they are not selected hereBoth, and they are not alternatives. The tests are well-formed and should run, so
backend_urlis defined rather than the module deselected — but every request goes
over the wire through a realhttpx.AsyncClient, so a live backend is a genuine
precondition and its absence is a skip with a stated reason.
knowledge_api_integration_test.pyis the contrast that settles it: it patches
aiohttp.ClientSession.get, needs only a valid URL string, and correctly just defines
the fixture.Nothing in #14933 narrows what the suite selects. Two tests assert that, on the
scheduled expression and on the dispatch default — the latter added because a
mutation of it slipped past the first version of the guard.- added 5 commits that reference this issue
on Aug 24, 2026 Result: the suite is green and reports a real count.
Run
32713266629on PR #14933's headb3096ed86— the first green run this workflow has ever had:35 passed, 49 skipped in 697.15sbefore (run 32702629146) after (run 32713266629) failed 19 0 errors 19 0 passed 35 35 skipped 12 49 collected 85 84 All four revised acceptance criteria are met. The 85 → 84 collection delta is not a lost test: the "before" run was on #14929's head, which adds
tools scripts pipeline-scripts autobot-infrastructure/shared/scripts/hooksto the pytest roots. That PR's own comment names the extra test —scripts/check_script_exec_bits_test.py'sintegrationcase. Nothing in #14933 removes a root, and it returns to the selection when #14929 lands.Two residuals filed rather than folded in:
- bug(test): the backend conftest Redis stand-in blocks every integration test from the real Redis the marker suite provides #14932 — the backend conftest Redis stand-in, the actual cause of the two knowledge-base failures. The suite provisions a healthy Redis that no backend test can reach.
- bug(vision): CLIPModel load is broken by a removed transformers kwarg, and the only test that notices flips on runner contention #14935 —
CLIPModelload broken by a removedtransformerskwarg, so no vision model loads at all; the only test that notices is a boundary flake that reddened one intermediate run at537.86msagainst a500.0msbudget. This one must be fixed before Marker-excluded tests (slow/integration/performance/distributed) run in no workflow at all #13286 enables the cron`, or the suite goes back to being red about half the time.
- added a commit that references this issue
on Aug 24, 2026 Delivered and merged as
74ba8f919(PR #14933).- CI: 50 success / 9 skipped / 0 failures / 0 pending, deduped to the latest run per check name with any queued or in-progress run counted as PENDING.
- Review: independent
code-reviewerpass, verdict clean, merge as-is.
before after failed 19 0 errors 19 0 passed 35 35 The 19 errors were one file, not 19 problems — a single test module whose conftest chain never reaches the only
backend_urldefinition. The 19 failures were three causes, and 35 of the 38 non-passes were the same shape: real HTTP to a service a GitHub runner does not have.The green is earned, not manufactured — the review checked this specifically, because skipped went 12 → 49 and that is exactly how a suite becomes green and worthless.
endpoint_is_listeningopens a real socket and catches onlyOSError; every other exception, including a caller bug, propagates. A malformed target raises rather than guessing a port. There is a test that injects aRuntimeErrorintosocket.create_connectionand asserts it is not swallowed — that is the one test that would catch a regression toexcept Exception. Every guarded module reaches the probe through an autouse fixture evaluated at runtime, so a runner that does have a live backend runs all of them for real.Narrowing is pinned in two independent places — one test asserts each marker name against the job-level
MARKER_EXPRESSION, another parses the YAML and asserts againstworkflow_dispatch.inputs.markers.default. Those are genuinely two different strings in the workflow, not one check wearing two hats.A genuine assertion defect was fixed: a benchmark asserted absolute process RSS < 500 MB while an xdist worker sits at ~1.6 GB from imports before the first benchmark runs. The budget was spent before the test started. It now measures the RSS delta across one mocked call, which a real per-iteration leak would still fail.
Four self-inflicted defects were caught and reported rather than hidden: a wiring test that passed vacuously because a renamed script matched no token; a guard that over-reached, skipping 7 tests where only 6 dial the backend; all 12 python-suite shards broken by a Python 3.14
@dataclassannotation-resolution change, fixed at source; and an unregistered env var.Deferred, not dodged — #14932. The workflow provisions, health-checks and configures a Redis that no backend test can reach:
autobot-backend/conftest.pyswapsautobot_shared.redis_clientfor a socket-free stand-in gated on directory membership and never on markers. The review confirmed the root cause independently. Fixing it flips Redis behaviour for all 35 currently-passing backend tests at once, which would make this PR's before/after comparison unattributable.One item carries a deadline: #14935.
CLIPModelload is broken by a removedtransformerskwarg, so no vision model loads at all. It must land before #13286 enables the cron, or this suite is red half the time again.Also filed from the review: #14941 —
test_celery_worker_statusreturns instead of asserting, so it cannot fail. Pre-existing, and noticed only because this PR had to read that test closely to decide it was correctly exempt from the liveness guard.- added a commit that references this issue
on Aug 24, 2026
The marker-excluded suite cannot run:
fixture 'backend_url' not foundExposed by PR #14929, not caused by it. That PR edits
.github/workflows/marker-tests.yml, and the workflow carriespull_request: paths: ['.github/workflows/marker-tests.yml']precisely so an edit runs it. Editing the file is what made the breakage visible.Why it is pre-existing
The failures are all
fixture 'backend_url' not found, raised from tests underautobot-backend— a root that was already in the invocation before #14929 touched it. #14929 only appendedtools scripts pipeline-scripts autobot-infrastructure/shared/scripts/hooks, none of which define or need that fixture.It is also not visible on
Dev_new_gui: the workflow is not apull_requestworkflow in the ordinary sense and does not run on base pushes — the last three base commits show no run of it at all. So this suite has been broken without any surface reporting it.Why it matters despite being non-blocking
The workflow's own header is explicit that a red run here is not an acceptable state:
That warning is now describing this file's own condition.
The suite exists to run the markers the PR gate excludes —
integration,slow,distributed,performance. While it cannot collect, every test carrying one of those four markers runs nowhere at all. That is the same class as #14884 and #14917: a suite that is configured, scheduled, and inert.Acceptance
backend_urlresolves for the marker-excluded run — either the fixture is provided in a conftest the run reaches, or the tests requiring a live backend are marked so they are not selected hereintegration/slow/distributed/performanceactually execute, so "collected zero" cannot read as "passed"Related: #13286 (owns the marker-excluded suite), #14884, #14917, #14891 — the same "configured but never runs" family.