Problem
autobot-backend/api/ide_integration_test.py::test_completion_performance ends with:
response = await engine.complete(sample_request)
# Should complete within reasonable time (< 200ms)
assert response.completion_time_ms < 200
completion_time_ms is pure wall clock — IDECompletionEngine.complete() computes it as
(_time.time() - start_time) * 1000. So the assertion is not a statement about the engine;
it is a statement that the machine was fast enough at that instant.
Under -n auto --dist loadscope it is not. Observed on a python-suite shard:
FAILED autobot-backend/api/ide_integration_test.py::test_completion_performance[asyncio]
- assert 1253.8769245147705 < 200
The PR that shard ran for changed no Python at all — its diff is frontend components, locale
files and a changelog entry. The test fails on load, not on a change, so it can redden any
branch that happens to share a loaded worker.
This is a known defect class in this exact file
#15861 fixed the same thing for the sibling tests and recorded the reasoning in the module
header of ide_integration_test.py:
A test that asserts on the ML path while letting the real clock run is asserting that this
machine was fast enough at that instant — which is why test_ml_completions failed on a PR
whose diff touched neither this module nor anything it imports.
That commit introduced _ControlledClock, already present in the file, and converted
test_ml_completions to it. test_completion_performance was left racing the clock.
Why not simply raise the threshold
Raising 200ms to some larger number buys time and keeps the defect: the assertion still
measures the host rather than the engine, and the number that makes it stop failing is the
number that makes it stop testing anything. A frozen clock alone is no better — it would make
the assertion vacuously true.
Acceptance criteria
Notes
The WARNING api.ide_integration: ML completion failed: 'CompletionTrainer' object has no attribute 'load_model' line in the same output is the fixture's ML path declining, which is
expected in this test — it is not the cause of the 1253ms and is not in scope here.
Problem
autobot-backend/api/ide_integration_test.py::test_completion_performanceends with:completion_time_msis pure wall clock —IDECompletionEngine.complete()computes it as(_time.time() - start_time) * 1000. So the assertion is not a statement about the engine;it is a statement that the machine was fast enough at that instant.
Under
-n auto --dist loadscopeit is not. Observed on apython-suiteshard:The PR that shard ran for changed no Python at all — its diff is frontend components, locale
files and a changelog entry. The test fails on load, not on a change, so it can redden any
branch that happens to share a loaded worker.
This is a known defect class in this exact file
#15861 fixed the same thing for the sibling tests and recorded the reasoning in the module
header of
ide_integration_test.py:That commit introduced
_ControlledClock, already present in the file, and convertedtest_ml_completionsto it.test_completion_performancewas left racing the clock.Why not simply raise the threshold
Raising 200ms to some larger number buys time and keeps the defect: the assertion still
measures the host rather than the engine, and the number that makes it stop failing is the
number that makes it stop testing anything. A frozen clock alone is no better — it would make
the assertion vacuously true.
Acceptance criteria
test_completion_performanceno longer reads the real clock; it uses the_ControlledClockalready in the file, on the
time.timethe engine resolves at call time.completion_time_msis the elapsedtime the engine actually measured — rather than the host's speed, and can still fail if
that reporting breaks.
_gather_completions' own latency gate is fix(api): the ML completion budget discards the result after paying its full latency #15863's subjectand is not touched here.
Notes
The
WARNING api.ide_integration: ML completion failed: 'CompletionTrainer' object has no attribute 'load_model'line in the same output is the fixture's ML path declining, which isexpected in this test — it is not the cause of the 1253ms and is not in scope here.