bug(test): 3 multimodal performance/scalability tests fail — metadata Mock lacks __setitem__ since #15234

Problem

autobot-backend/system_benchmarks_performance_test.py has 3 failing tests on main, all with the same root cause:

FAILED system_benchmarks_performance_test.py::TestSystemPerformanceBenchmarks::test_multimodal_processor_performance - assert False is True
FAILED system_benchmarks_performance_test.py::TestSystemPerformanceBenchmarks::test_concurrent_processing_performance - assert False
FAILED system_benchmarks_performance_test.py::TestScalabilityBenchmarks::test_multimodal_processor_scalability - assert False

Underlying error, logged from multimodal_processor.processor:processor.py:173:

ERROR multimodal_processor.processor:processor.py:173 Multi-modal processing failed: 'Mock' object does not support item assignment

Root cause: processor.py:166 (added for #15234 — "the caller could not tell a persisted result from a dropped one") writes:

result.metadata["persisted"] = await self._store_result(result)

The three failing tests mock context_processor.process's return value as a plain Mock(...) (from unittest.mock import AsyncMock, Mock, patch, not MagicMock) with no metadata= kwarg set — e.g. system_benchmarks_performance_test.py:184-191:

mock_process.return_value = Mock(
    success=True, confidence=0.8, processing_time=0.1,
    result_data={"decision": "test"}, modality_type=ModalityType.TEXT,
    intent=ProcessingIntent.DECISION_MAKING, result_id="test",
)

result.metadata therefore resolves to an auto-generated child Mock, and plain Mock (unlike MagicMock) does not implement __setitem__ — so result.metadata["persisted"] = ... raises, process()'s except-block catches it, and the result comes back with success=False, which is what each test's assert result.success is True / assert all(r.success for r in results) then fails on.

Found while reviewing #16801 (the main → release promotion PR): its "Run slow / integration / distributed / performance tests" CI job is red because of this. These tests carry the slow/integration/distributed/performance markers, which don't run on a normal PR's default CI — so this has likely been broken on main for a while without being caught; #15234 is the probable introduction point.

Fix

Either add metadata={} (or a real dict) to each affected Mock(...) construction in system_benchmarks_performance_test.py (the 3 call sites feeding the failing tests, at minimum lines ~184, ~221, ~537 — verify each is actually reached by one of the 3 failing tests before touching it), or switch those constructions from Mock to MagicMock so __setitem__ works as the real ProcessingResult/dict-like object would.

Acceptance criteria

python3 -m pytest -m "slow or integration or distributed or performance" system_benchmarks_performance_test.py — 0 failures related to this root cause.
The fix touches only the test fixtures (or confirms a production-code issue instead, with evidence) — this is a test/mock gap, not a request to revert bug(multimodal): _store_result swallows every exception, including the tenancy rejection it was firing on continuously #15234's behavior.
No other Mock(...) (vs MagicMock) construction in this file has the same latent gap against a result.metadata[...] write elsewhere in processor.py.

AC	Evidence on `main`	Verdict
1. The marker suite passes (`-m "slow or integration or distributed or performance"`)	`marker-tests.yml` runs this expression on pushes to `main`. Its run for the merge commit has not finished. The last completed `main` run (`928573e3b`, from before this fix) failed.	pending
2. The fix touches only test fixtures	#16995 changes exactly one file: `autobot-backend/system_benchmarks_performance_test.py`	met
3. No other `Mock(...)` in the file has the same gap	The file has two `Mock(` sites. Line 74 is the new `_mock_processed_result()` helper, which sets `metadata={}`. Line 435 feeds only `processor._update_stats(result)`. The only `result.metadata[...]` write in `multimodal_processor/processor.py` is line 167, on the `process()` path.	met

This closes again once marker-tests passes on a main commit that contains 33481c0b3e, with no failures from this root cause.

Chunking triage — a proposal, not an assignment

Proposed priority: priority: medium — not applied.
triage(v0.9.0): release criticality of the 123 open issues — 26 blocking, 36 small, 18 umbrellas, 41 misfiled, 2 undetermined #17639 criterion met: none of the five. Placed as scoped and small: one PR, no open question, not blocking.
triage(v0.9.0): release criticality of the 123 open issues — 26 blocking, 36 small, 18 umbrellas, 41 misfiled, 2 undetermined #17639 bucket: 2 — scoped and small (one PR, no open question, not blocking)
Umbrella / container: no.
Pre-filter: ⚠ a merged commit references this issue — faa476f467 refactor(test): extract a shared mock-result helper. A reference is not a delivery: verify AC coverage before putting it in a chunk, and consider a closure pass first.
Basis: triage(v0.9.0): release criticality of the 123 open issues — 26 blocking, 36 small, 18 umbrellas, 41 misfiled, 2 undetermined #17639's per-issue row, reused rather than re-derived — "No code left. delivered: prior cites marker job passing at 6680ed7 with fix on main and AC2/AC3 verified - only closure remains"

Nothing was relabelled, moved or closed by this pass.

Closure pass — 2 of 3 criteria met from merged code; the third needs a test run I cannot do here. Not closed.

Evidence at origin/main 7adaca8c29. File: autobot-backend/system_benchmarks_performance_test.py (note: not under multimodal/).

AC2 — the fix touches only the test fixtures — MET. The change is a fixture argument with its reason recorded at :71-73:

"#16990: metadata={} is required — processor.py writes result.metadata["persisted"] = … (#15234), and a plain Mock's auto-generated .metadata has no __setitem__."

and metadata={} passed at :82. No production code is implicated, which is what this criterion asked to be confirmed either way.

AC3 — no other Mock(...) construction in the file has the same latent gap — MET, and by a stronger measure than the criterion asked. There are zero MagicMock occurrences and no remaining bare Mock( / Mock() constructions in the file, so no other site can hit a result.metadata[...] write with an auto-generated attribute. The class of defect has no second instance here.

AC1 — the marked pytest selection shows 0 related failures — NOT MET, and not measurable by me. Running the suite means executing repository code, which the current owner ruling reserves to the hooks and CI. This one needs the evidence to come from a CI run: -m "slow or integration or distributed or performance" over that file, with the output pasted.

Not closed. The fix and its blast radius are verified in code; the confirming run is the whole remainder, and it is one CI invocation.

`v0.9-host` review — this issue is not waiting on a host. Its remaining criterion is a CI run.

Part of a review of all ten v0.9-host issues: for each, is host evidence genuinely the only thing missing, or is something else hiding behind "waiting for a deployment"?

This one is in the wrong place. Its open criterion needs a run, not a deployed host — the distinction matters because the deployment call is an event nobody controls, while a CI run is available today. Parked here, it waits for the wrong thing.

Detail is in my earlier per-AC audit on this issue. The short form:

bug(test): 3 multimodal performance/scalability tests fail — metadata Mock lacks __setitem__ since #15234 #16990 — AC1 is pytest -m "slow or integration or distributed or performance" over system_benchmarks_performance_test.py with the output pasted. A CI invocation. (AC2 and AC3 are already verified in code.)
feat(slm): Data hygiene page — list, preview and confirm cleanup of orphaned data; host #16927's unreachable-resource repair (#17038) #17040 — AC4's "all strings are i18n'd" is an absence claim, and the only thing that settles it is the repository's own hardcoded-UI-string guard. A guard run. (Locale coverage is already 11 of 11; the other four criteria are met.)
ci(docker): an unreachable Ubuntu mirror surfaces as a misleading 'held broken packages' error; apt steps neither retry nor fail on index errors #16291 — AC4 wants a build log showing the retry or the explicit apt-get update failure under a simulated unreachable mirror. That is a build with a manipulated network, not a deployment — harder than a plain CI job, and still not the deployment call. (The other three are verified in code.)

Recommendation: move these three out of v0.9-host. They belong wherever a "needs a run" bucket lives, or back in their topic chunk with the criterion marked needs a CI run. Leaving them here means the deployment call arrives, nobody looks at them because they are not host work, and they stay parked for the next reason.

Verdict: can be run today — and the code fix already landed

Freed from the host-blocked parking milestone: this never needed a deployment.

The job exists. .github/workflows/marker-tests.yml is on main:

Evidence	Where
Job name — literally this AC's test set	`:154` `name: Run slow / integration / distributed / performance tests`
`workflow_dispatch` input default	`:128` `default: 'integration or slow or distributed or performance'`
Expression the job runs	`:176` `MARKER_EXPRESSION: ${{ github.event.inputs.markers \|\| 'integration or slow or distributed or performance' }}`
Nightly cron — commented out	`:122` `# - cron: '20 3 * * *'`, gated on "#13286's closing step once #15054 lands"
Auto triggers today	`push`/`pull_request` scoped to `paths: ['.github/workflows/marker-tests.yml']` only

So the remaining AC needs one manual dispatch, not a new job.

The fix is already on main — commit faa476f467 (2026-09-18), which carried this issue's number while extracting the helper for a file-size ratchet trip:

autobot-backend/system_benchmarks_performance_test.py:69-82 — _mock_processed_result() returns Mock(..., metadata={}, ...), with the comment at :71-73 naming this issue and processor.py's result.metadata["persisted"] = ... (bug(multimodal): _store_result swallows every exception, including the tenancy rejection it was firing on continuously #15234) as the reason.
All three tests named in the report now take their mock from it: :201 (test_multimodal_processor_performance), :230 (test_concurrent_processing_performance), :538 (test_multimodal_processor_scalability).

The second AC's "touches only the test fixtures" holds — the change is the fixture helper, processor.py is untouched.

Correction to a related claim: #13286 states marker-excluded tests run in no workflow. That was true when written and is stale — marker-tests.yml landed 2026-09-14. What remains true is that no scheduled run exists, because the cron is still commented out.

Not ticking the first AC myself: it asks for a run's output, and I have no run. A dispatch of marker-tests.yml against main produces it.

The code fix is already on main; the open criterion is purely evidential. faa476f467 (2026-09-18) carries it — a file-size-ratchet extraction whose subject does not name this issue, which is why a subject scan misses it. autobot-backend/system_benchmarks_performance_test.py:69-82 _mock_processed_result() returns Mock(..., metadata={}, ...) with :71-73 citing this issue and #15234 by number; all three named tests draw from it (:201, :230, :538).

So AC1 asks for a run's output, and the run exists: .github/workflows/marker-tests.yml is workflow_dispatch-able (:123) with the default marker expression 'integration or slow or distributed or performance' (:128). One dispatch against main produces the evidence. Not ticked here — nobody has the output yet, and a criterion asking for a paste is not met by knowing it would pass.

Labelled `blocked: observation-gated` — AC1 names a run that cannot happen today

AC2 is ticked. AC1 is:

python3 -m pytest -m "slow or integration or distributed or performance" system_benchmarks_performance_test.py — 0 failures related to this root cause.

That marker set has no reachable CI run. .github/workflows/marker-tests.yml is the workflow that executes it, and its own header states the position at :75:

# SCHEDULE STILL WITHHELD — and #13286 stays open for exactly one reason.
# A permanently red scheduled workflow is not a signal; it is noise that trains …

So the suite is deselected on the PR gate and its scheduled runner is deliberately not enabled. The criterion asks for an outcome from a run nobody can trigger.

And it is not satisfiable locally either, which is why this is blocked rather than merely awkward. These are multimodal performance and scalability tests exercising production code, not AST-only guards — the standing rule is that a session verifies via CI or the deployed host, not by running codebase behaviour. The one lane that is open (repo_tests and --check scripts on the /opt interpreter) does not cover this.

So the dependency is #13286, and it is recorded nowhere on this issue. This is the second issue found tonight in exactly that position — #15055 asks for "ten consecutive scheduled runs green, cited by run id" against the same withheld schedule, where 19 of the last 20 runs are pull_request and the single schedule did not pass.

Two issues, one blocker. Whoever rules on #13286 should know it gates these, and both should carry a blocked_by edge rather than reading as ordinary open work.

AC1 is already satisfied by an existing run — no dispatch needed, and the `blocked_by` edge looks unjustified

Three findings, in the order they change the disposition.

1. The root cause is fixed, and by a better fix than the criteria describe

autobot-backend/system_benchmarks_performance_test.py:71-73 carries the fix and names this issue:

processor.process() benchmarks below. #16990: metadata={} is required — processor.py writes result.metadata["persisted"] = … (#15234), and a plain Mock's auto-generated .metadata has no __setitem__.

With metadata={} at :82 and metadata={"result": …} at :288. Note it is not a Mock → MagicMock swap — the file still has two Mock( and zero MagicMock(. Passing a real dict is the stronger fix: MagicMock would have absorbed the __setitem__ silently and the benchmark would no longer exercise the write that #15234 introduced.

2. AC1 is satisfied by run `35862044096`, with a control

AC1 asks for pytest -m "slow or integration or distributed or performance" system_benchmarks_performance_test.py with 0 failures from this root cause. marker-tests.yml has five pull_request runs, all success, and the most recent two cover this file:


run `35862044096` (2026-09-23)	mentions `system_benchmarks_performance` 13 times; results `10 passed`, `53 passed`
run `35849499525`	identical

The control, because a zero from a log grep is usually a broken grep: the log fetches at 2081 lines, and failed (word-boundary, case-insensitive) occurs once — inside a summary table's column header, | Invocation | Collected | passed | failed |. Not a test failure. So the absence of failures is a measurement rather than a quiet grep.

3. The `blocked_by` edge to #13286 is not justified by this issue's own body

This issue carries a live blocked_by edge to #13286 (the withheld marker schedule). Its body does not support that: the only relevant passage, at :35, says the defect was "found while reviewing #16801 … its 'Run slow / integration / distributed / performance tests' CI job is red because of this", and goes on to note the markers "don't run on a normal PR's default CI". Nothing here depends on a scheduled run — AC1 names a pytest invocation, and a pull_request run of the marker workflow satisfies it as written.

That matters for whether this ever gets picked up: an issue whose single remaining criterion is already met, sitting behind an edge to an issue that cannot close until a cron is enabled, will never be selected by any pass that filters on open blockers.

Disposition

AC2 and AC3 are ticked. AC1's evidence is above. If the edge to #13286 is genuinely unintended, removing it and closing on run 35862044096 is the whole of the remaining work. If someone decided the marker suite must run on a schedule for this to count, that is a stricter reading than AC1's wording and should be said on the issue rather than carried as an edge.

Not ticking or closing — the evidence is here for whoever does, and the edge is not mine to remove.

fix(ansible): verify the repo on the fresh-add path, not only when on…

Closing — all three criteria met. The evidence for the last one has existed since 23 September and nobody looked.

AC1 — a marker-selected run with no failures from this root cause

Run 35862044096, workflow Marker-excluded Tests, on main, 2026-09-23, conclusion=success. Its step is named "Run slow / integration / distributed / performance tests" — the marker selection AC1 asks for.

Verified against the log myself rather than taken on report:

check	result
mentions of `system_benchmarks_performance`	13
occurrences of the word `failed` in 2,081 lines	1
...and that one is	a column header: `\| Invocation \| Collected \| Executed \| Passed \| Failed \| Errors \| Skipped \|`
pytest summaries	`10 passed, 24 skipped` · `53 passed, 76 skipped`

The control matters here, because a zero from a log grep is usually a claim about the grep: the single failed hit being a table header is what makes the absence a measurement rather than a miss.

AC2 and AC3 — already ticked, and the fix is better than the criteria asked

system_benchmarks_performance_test.py:71-73 cites this issue and uses metadata={} — a real dict — rather than swapping Mock for MagicMock. That is the stronger fix and worth recording why: a MagicMock would have absorbed the __setitem__ silently, the benchmark would have stopped exercising #15234's write, and every test would still have passed. The file retains two Mock( constructions and zero MagicMock(, which is AC3 holding.

A note for whoever maintains #13286

This issue carries a blocked_by edge to #13286 that #13286's body does not justify — nothing here depends on a scheduled run, only on a marker-selected one, and pull_request runs of that workflow have existed and passed for ten days. #13286's own title ("marker-excluded tests run in no workflow at all") is itself stale: the workflow exists with five successful runs; what is missing is the cron.

So this issue sat closeable behind a blocker whose premise had changed, invisible to any sweep that filters on open blockers. The edge is #13286's owner's to cut — I am not touching it — but it no longer blocks anything here. #15055's edge to the same blocker is justified, since only the cron produces scheduled runs; same blocker, one sound edge and one not.

Found by @autobot-ai-ae, who checked the run list instead of trusting a note saying a workflow_dispatch was needed. That note was mine and it was wrong.

fix(ansible): the fresh-install path added the repo and never verifie…

Navigation Menu

Repository navigation

bug(test): 3 multimodal performance/scalability tests fail — metadata Mock lacks setitem since #15234 #16990

Problem

Fix

Acceptance criteria

Related

Chunking triage — a proposal, not an assignment

Closure pass — 2 of 3 criteria met from merged code; the third needs a test run I cannot do here. Not closed.

`v0.9-host` review — this issue is not waiting on a host. Its remaining criterion is a CI run.

Verdict: can be run today — and the code fix already landed

Labelled `blocked: observation-gated` — AC1 names a run that cannot happen today

AC1 is already satisfied by an existing run — no dispatch needed, and the `blocked_by` edge looks unjustified

1. The root cause is fixed, and by a better fix than the criteria describe

2. AC1 is satisfied by run `35862044096`, with a control

3. The `blocked_by` edge to #13286 is not justified by this issue's own body

Disposition

AC1 — a marker-selected run with no failures from this root cause

AC2 and AC3 — already ticked, and the fix is better than the criteria asked

A note for whoever maintains #13286

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

bug(test): 3 multimodal performance/scalability tests fail — metadata Mock lacks __setitem__ since #15234 #16990

Description

Problem

Fix

Acceptance criteria

Related

Activity

mrveiss commented on Sep 19, 2026

mrveiss commented on Sep 28, 2026

Chunking triage — a proposal, not an assignment

mrveiss commented on Sep 28, 2026

Closure pass — 2 of 3 criteria met from merged code; the third needs a test run I cannot do here. Not closed.

mrveiss commented on Sep 29, 2026

v0.9-host review — this issue is not waiting on a host. Its remaining criterion is a CI run.

mrveiss commented on Sep 29, 2026

Verdict: can be run today — and the code fix already landed

mrveiss commented on Sep 29, 2026

mrveiss commented on Oct 3, 2026

Labelled blocked: observation-gated — AC1 names a run that cannot happen today

mrveiss commented on Oct 3, 2026

AC1 is already satisfied by an existing run — no dispatch needed, and the blocked_by edge looks unjustified

1. The root cause is fixed, and by a better fix than the criteria describe

2. AC1 is satisfied by run 35862044096, with a control

3. The blocked_by edge to #13286 is not justified by this issue's own body

Disposition

mrveiss commented on Oct 3, 2026

AC1 — a marker-selected run with no failures from this root cause

AC2 and AC3 — already ticked, and the fix is better than the criteria asked

A note for whoever maintains #13286

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

bug(test): 3 multimodal performance/scalability tests fail — metadata Mock lacks setitem since #15234 #16990

`v0.9-host` review — this issue is not waiting on a host. Its remaining criterion is a CI run.

Labelled `blocked: observation-gated` — AC1 names a run that cannot happen today

AC1 is already satisfied by an existing run — no dispatch needed, and the `blocked_by` edge looks unjustified

2. AC1 is satisfied by run `35862044096`, with a control

3. The `blocked_by` edge to #13286 is not justified by this issue's own body