Repository navigation
Marker-excluded tests (slow/integration/performance/distributed) run in no workflow at all #13286
Description
Activity
- added 4 commits that reference this issue
on Aug 2, 2026 Progress from PR #13346 (merged as
6eca30d25) — still open, deliberately.marker-tests.ymlnow has a trigger that fires, and completed its first run in its existence:= 11 failed, 7 passed, 1 skipped, 2320 warnings, 40 errors in 663.05s (0:11:03) =plus
1 erroron the slm leg (collection).Before this,
gh run list --workflow=marker-tests.ymlanswered "could not find any workflows named marker-tests.yml" — GitHub had never registered it, becausescheduleandworkflow_dispatchregister from the default branch only and the file exists only onDev_new_gui. Tracked as #13352.So the marker-excluded suite has now run once, and it is red. The 40 errors are dominated by collection failures under
repo_tests/lint/canonical/**; the 11 failures are mostlyapi/voice_integration_test.py(wants a live TTS worker) plustest_autobot_agent_adapter.py::test_integration_summarization_agent_reaches_completed.This issue's own text says a nightly that is red from day one is the same defect class it exists to fix, so it does not close here. What remains:
- Promote the workflow to
mainso the schedule can actually fire (ci: scheduled workflows are dormant — they exist only on Dev_new_gui, and main runs a broken health check every 30min #13352) — until then the added triggers arepaths-scoped to the workflow file itself and fire once on merge, then never again. - Triage the first run: the
lint/canonicalcollection errors are being fixed independently in PR fix(tests): clear all collection-error clusters polluting every python-suite shard (#13086) #13349; the live-TTS-worker tests need either a service or a marker.
Only then does the marker-excluded suite have a cadence, which is what this issue asks for.
- Promote the workflow to
#13391 withholds this workflow's nightly cron ahead of the
mainpromotion (#13389).Context: the cron has never fired — GitHub registers
schedule/workflow_dispatchfrom the default branch only, and this file exists solely onDev_new_gui, so the workflow has produced zero runs of any kind. Promotion would register it immediately, and its one observed run is red (11 failed, 7 passed, 40 errors). That would install a nightly failing every night against the default branch.The cron is commented, not deleted, with the restore condition recorded inline.
workflow_dispatchstays live so the suite remains runnable on demand while the 51 failures are triaged.Uncommenting the
schedule:block is the closing step of this issue — please do not close #13286 without it, or the nightly this issue exists to deliver still will not run.- added a commit that references this issue
on Aug 3, 2026 Adding one file to this issue's triage list, from #13551 batch C (PR #13566).
autobot-backend/knowledge/kb_optimization_test.py::test_knowledge_base_optimization
is newly markedintegration, so as of that PR it runs onworkflow_dispatch
only — i.e. nowhere automatically, which is exactly what this issue is about.Why it was moved rather than fixed in place: it needs a live Redis
knowledge
database plus ChromaDB. Without themget_knowledge_base()raises
RuntimeError: Failed to initialize knowledge base, so it cannot be made
hermetic. It was failing on the PR unit gate for that reason. Coverage impact is
nil — the body contains no assertions at all; it prints and returns a dict, so
it could only ever have failed on infrastructure, never on a regression.Two things to know before this suite is switched on, so neither is a
surprise on the first green attempt:-
The test needs the
knowledgeRedis database and a ChromaDB instance
provisioned in whatever job ends up running the excluded markers. The
redis:7-alpineservice already declared inmarker-tests.ymlis necessary
but not sufficient — the KB resolves a namedknowledgedatabase, and
ChromaDB is separate. -
It returns a dict instead of asserting. Worth cleaning up as part of making
this suite meaningful — a test that asserts nothing cannot fail usefully.One correction on that second point, since it came up in review of fix(backend): repair threat analyzers, watch-folder ingest and three tests (#13551) #13566 and
is easy to get wrong: this is not a hard failure on the current stack. I
checked it on the CI-parity venv (build(dev): give local gates the same Python CI runs (#13573) #13574 — Python 3.14.6, pytest 9.1.1,
pytest-asyncio 1.4.0,pytestunpinned inrequirements-ci-test.txt:14):- a sync test returning non-
NoneraisesPytestReturnNotNoneWarning,
a warning, and still passes; - an async test returning non-
None— which this one is — produces no
warning at all, because pytest-asyncio consumes the return value.
So it is a code smell to fix on its merits, not a blocker that will trip the
suite. Recording it here so nobody budgets for a failure that will not
happen, and so the real prerequisite (item 1) gets the attention. - a sync test returning non-
-
- added a commit that references this issue
on Aug 4, 2026 10 remaining items
Correction — the schedule is NOT on, and this issue stays open
My previous comment said the cron was restored as the closing step. A third run
(32839088246) makes that wrong, and the honest thing is to withdraw it rather than let a
green-looking claim stand.What the third run showed
invocation collected passed failed errors skipped backend 85 34 1 0 50 slm 0 0 0 0 0 infra 12 10 0 0 2 test_celery_worker_statusnow skips correctly. But the one remaining failure is
system_benchmarks_performance_test.py::test_system_startup_performance, which passed on run
32837489748 and failed on 32839088246 — minutes apart, no code change,547msagainst a
hardcoded500msbudget. It asserts wall-clock milliseconds on a shared runner, so it turns on
runner load. Filed as #15055.Underneath it, #15054:
resume_downloadwas removed in the pinnedtransformers==5.15.0,
and all ten transformers call sites in the backend still pass it, so CLIP loading raises
TypeErrorand the startup being timed is an error path. That has to be fixed before the timing
means anything.Decision
The cron stays commented, and
marker-tests.ymlrecords exactly what unblocks it. Turning it on
over an assertion that flips on runner weather would produce the permanently-red-and-therefore-
ignored scheduled workflow this issue exists to prevent — theself-hosted-runner-healthhabit
the file's own comment warns against.MARKER_MIN_PASSEDis set to 34, the lower of the two observed figures, with the reason
recorded in the workflow. It goes to 35 when #15055 lands and the number stops moving.What IS done
- All 95 marker-carrying tests are named by a workflow root; the 12 that ran nowhere now run.
- Every defect the new trees surfaced was fixed, not excluded — 5 failures and 1 collection
error down to 0. - Floors are per invocation, not on the union, with
slm=0backed by an oracle. repo_tests/marker_suite_root_coverage_test.pyfails if a marked module ever lives under no
root again.
Remaining for closure: #15054, then #15055, then turn the cron on and raise the floor.
The suite is green — run 32840740886
invocation collected passed failed errors skipped collected floor passed floor backend 85 35 0 0 50 1 34 slm 0 0 0 0 0 0 (declared) 0 (declared) infra 12 10 0 0 2 1 10 (declared) total 97 45 0 0 52 First green run this suite has ever had, and the whole of PR #15048 is green with it
(64 pass / 11 skipping / 0 fail).The cron still stays off. A green run does not undo a flake proven by two contradicting runs
on identical code —test_system_startup_performancepassed on 32837489748, failed on
32839088246 at 547ms against a 500ms constant, and passed again here. Turning the schedule on now
would bet this workflow's credibility on a coin flip, which is the precise failure mode this
issue exists to prevent. #15055 first, then #15054's error path underneath it, then the cron and
the floor at 35.- added a commit that references this issue
on Aug 25, 2026 Stale-issue sweep (never auto-closed — Dev_new_gui wasn't GitHub's default branch at merge time). Verified against current
origin/main; leaving open.The mapped PR (#13391) delivered exactly the interim mitigation it says it would:
.github/workflows/marker-tests.yml:118-121— theschedule:/cronblock is commented out,workflow_dispatch:is live at :122. Its own body states: "Restore condition, for whoever closes #13286: uncomment the schedule: block once the 11 failures and 40 collection errors are green."The core ask (a live scheduled run of marker-excluded tests) is still not in place: the current file's own header comment (line 74) states "SCHEDULE STILL WITHHELD — and #13286 stays open for exactly one reason." A later comment (line 39) notes the underlying failures were root-caused/fixed as of 2026-08-24 via #14930, but the schedule itself was never re-enabled. Grep across
.github/workflows/*.ymlconfirms no other workflow runs theslow/integration/distributed/performancemarkers on any trigger.- added a commit that references this issue
on Sep 12, 2026 Status check on the cron-enablement blockers. #15054 (the first blocker) is closed. The second (ten consecutive green runs) is genuinely not met — checked real run history (
gh run list --workflow=marker-tests.yml), and most recent runs fail on a currently-live, previously-untracked defect:Multimodal processor startup's performance budget was calibrated while #15054's bug was silently skipping the real work, never re-derived after #15297 fixed it (exactly the "baseline's premise had gone" patternperf_work_budget.py's own docstring already describes). Filed as #16535 with 7 real measurements and fixed via PR #16537.Once that merges, the suite should start accumulating real green runs toward the ten-in-a-row bar — this comment is a status update, not a claim that the cron can be turned on yet.
This issue's central claim is stale and anything routing off it needs re-reading first. The title says marker-excluded tests run in no workflow. True when filed; false since
.github/workflows/marker-tests.ymllanded (verified onorigin/main::72name: Marker-excluded Tests,:123workflow_dispatch:live,:128default marker expression'integration or slow or distributed or performance',:154the job).What remains true is narrower: no scheduled run exists. The cron at
:119-122is deliberately commented out, and the file states why at:77— "A permanently red scheduled workflow is not a signal; it is noise that trains…" — with:102recording the condition for turning it on: "cron is not turned on against an argument — it is turned on against" evidence. The auto triggers arepush/pull_requestscoped topaths: ['.github/workflows/marker-tests.yml']only, i.e. the workflow tests itself and nothing else.So this issue should be re-scoped to "the marker suite has no scheduled run, and the condition for enabling its cron is unmet", naming what that condition is. Filed as-is, it reads as no workflow exists and will keep sending people to build one.
Found while auditing #16990, whose remaining AC is satisfiable by one dispatch of this workflow.
This issue's own premise has gone stale, and one of its two
blocked_byedges can be cutThis issue blocks #15055 and #16990. One of those edges is justified and one is not, and the title no longer describes the state of the tree.
The title is stale: the workflow exists and has run
"Marker-excluded tests … run in no workflow at all" was true when filed.
.github/workflows/marker-tests.ymlnow exists, and has fivepull_requestruns, allsuccess— most recently35862044096and35849499525on 2026-09-23.What is actually missing is the cron, and the workflow says so in its own words at
:75:# SCHEDULE STILL WITHHELD — and #13286 stays open for exactly one reason.with
:122holding the commented-out- cron: '20 3 * * 0'and:114naming the condition for enabling it. Thepushandpull_requesttriggers are filtered topaths: ['.github/workflows/marker-tests.yml'](:139-143), so the workflow runs only when its own file changes — which is why runs exist and nobody noticed them.That distinction matters for anything reasoning about this issue: "no workflow" and "a workflow with no schedule" license different conclusions, and the second is the true one.
The edge to #16990 is not justified by #16990's body — cutting it closes that issue
#16990's single open criterion is a pytest invocation:
pytest -m "slow or integration or distributed or performance" system_benchmarks_performance_test.pywith 0 failures from its root cause. Nothing in it asks for a scheduled run. Its only relevant passage says the defect was found because #16801's marker job was red and that these markers "don't run on a normal PR's default CI".Run
35862044096satisfies that criterion: it mentionssystem_benchmarks_performance13 times and reports10 passed/53 passed. The control, because a zero from a log grep is usually a broken grep — the log fetches at 2,081 lines andfailedoccurs once, inside a summary table's column header (| Invocation | Collected | passed | failed |). No test failure.So #16990's last criterion has been met for ten days, and it sits behind this edge where no selection pass filtering on open blockers will ever surface it. If that edge was unintended, removing it closes #16990 on
35862044096. Evidence is on that issue asissuecomment-5971246384. I have not touched the edge — it is yours to cut, and an edge recording a real past constraint is history worth keeping rather than tidying away.The edge to #15055 is justified, and that issue needs a different remedy
#15055 asks for ten consecutive green scheduled runs, and only the cron produces those — so the dependency is real. But this workflow also records, at
:105:# #15055 IS FIXED. system_benchmarks_performance_test.py no longer compares [absolute wall-clock]So #15055's defect is fixed and only its verification criterion remains, and that criterion counts a population of zero: there have never been any scheduled runs, and this issue is one of the stated reasons for withholding the schedule. Ten greens is unreachable, not slow — leaving it unticked reads as "nobody got round to it" when the truth is "no run of that kind can exist yet".
That argues for restating #15055's criterion as the property (the benchmarks are not flaky, asserted against runs that actually occur) rather than as a count of runs on a trigger that is deliberately off.
What would close this issue
This has no acceptance criteria of its own. Given the above, the closing condition is narrower than the title suggests: enable the cron, which
:114gates on #15054 landing. The two dependents want different things from it — #16990 wants nothing, and #15055 wants a run count that only the cron can produce.
What
Nothing in
.github/workflows/ever runs a test markedslow,integration,distributedorperformance. Those markers are a one-way exit from CI, not a routing decision.Evidence
ci.ymlis the only workflow that runs the Python suite, and both of its invocations filter the markers out:Grepping every workflow for a marker selection returns only those two exclusions — no
-m "slow", no-m "integration", nowhere.The eleven scheduled workflows are
branch-cleanup,branch-health-report,canonical-audit,codeql,dependabot-auto-merge,docker-smoke-test,hardened-smoke-test,security,self-hosted-runner-health,stale-branches-warning,unwired-tracker-audit. The only one that invokes pytest at all runs a single file (unwired-tracker-audit.yml:60,pipeline-scripts/test_audit_unwired_trackers.py).ci.ymldoes have a "Run integration tests" step, but it is directory-based (infrastructure/shared/tests/integration), not marker-based — marking a testintegrationdoes not route it there.Why it matters
Every
slow/integration/performancemark added to date silently retired that test. #13284 adds four more (three of which assert nothing, so they cost no coverage — butsecurity_edge_cases_test.py::test_maximum_command_lengthdoes assert, and it now runs nowhere).The gate-promotion decision on #10691 also reads differently depending on whether marker-excluded tests have a home.
Scope
Add a scheduled workflow that runs the excluded markers — e.g. nightly,
-m "slow or integration or performance", no coverage gate, failures reported rather than blocking. Note the self-hosted runner is a singleton, so it needs a schedule slot that does not collide with the PR gate.Alternatively, decide explicitly that marker-excluded tests are retired and record that, so future marking is an informed choice.
Found while triaging #13284.