Skip to content

test(api): test_completion_performance races the real clock, the defect #15861 already fixed in its siblings #16928

Description

@mrveiss

Problem

autobot-backend/api/ide_integration_test.py::test_completion_performance ends with:

response = await engine.complete(sample_request)

# Should complete within reasonable time (< 200ms)
assert response.completion_time_ms < 200

completion_time_ms is pure wall clock — IDECompletionEngine.complete() computes it as
(_time.time() - start_time) * 1000. So the assertion is not a statement about the engine;
it is a statement that the machine was fast enough at that instant.

Under -n auto --dist loadscope it is not. Observed on a python-suite shard:

FAILED autobot-backend/api/ide_integration_test.py::test_completion_performance[asyncio]
  - assert 1253.8769245147705 < 200

The PR that shard ran for changed no Python at all — its diff is frontend components, locale
files and a changelog entry. The test fails on load, not on a change, so it can redden any
branch that happens to share a loaded worker.

This is a known defect class in this exact file

#15861 fixed the same thing for the sibling tests and recorded the reasoning in the module
header of ide_integration_test.py:

A test that asserts on the ML path while letting the real clock run is asserting that this
machine was fast enough at that instant — which is why test_ml_completions failed on a PR
whose diff touched neither this module nor anything it imports.

That commit introduced _ControlledClock, already present in the file, and converted
test_ml_completions to it. test_completion_performance was left racing the clock.

Why not simply raise the threshold

Raising 200ms to some larger number buys time and keeps the defect: the assertion still
measures the host rather than the engine, and the number that makes it stop failing is the
number that makes it stop testing anything. A frozen clock alone is no better — it would make
the assertion vacuously true.

Acceptance criteria

  • test_completion_performance no longer reads the real clock; it uses the _ControlledClock
    already in the file, on the time.time the engine resolves at call time.
  • The assertion pins a property of the engine — that completion_time_ms is the elapsed
    time the engine actually measured — rather than the host's speed, and can still fail if
    that reporting breaks.
  • No production code changes. _gather_completions' own latency gate is fix(api): the ML completion budget discards the result after paying its full latency #15863's subject
    and is not touched here.

Notes

The WARNING api.ide_integration: ML completion failed: 'CompletionTrainer' object has no attribute 'load_model' line in the same output is the fixture's ML path declining, which is
expected in this test — it is not the cause of the 1253ms and is not in scope here.

Activity

  1. added this to the v0.9.0 milestone on Sep 18, 2026
  2. added a commit that references this issue on Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions