Skip to content

perf(tests): four guard modules cost 64s at import time, and every changeset touching repo_tests/_reach.py pays all of it #16188

Open

Description

@mrveiss

Four guard modules spend 64 seconds at IMPORT time, before a single test runs

Measured on origin/Dev_new_gui by importing each repo_tests/*_test.py and timing
the import alone — no test execution:

module import time
no_duplicate_dict_keys_test 25.5s
man_page_indexer_path_resolves_15853_test 24.5s
model_pricing_tables_agree_15912_test 11.4s
reach_declarations_test 2.3s
total 63.6s

Importing the full guard set takes 67s. So 95% of it is these four, and the
work is at module scope rather than inside a test.

One of them starts a live client during import:

llm_shared.mock_providers - INFO - LocalLLM initialized with Ollama at http://127.0.0.1:11434

Why this is a real cost and not a curiosity

repo_tests/reach_declarations_test.py imports every guard module to enumerate
declare() calls. Any change to repo_tests/_reach.py therefore pays all 67
seconds before running one assertion — against a 128-second pre-push budget.

That is the mechanism behind the timeouts on #15928: not that reach discovery is
expensive (measured at 0.02–0.04s per declaration), but that importing the
guards to find the declarations costs 67s.

The shape

Import time is a cost every consumer pays, whether or not it uses the expensive
thing. A test module that builds its fixture at module scope charges that to:

  • the pre-push hook, for any changeset that selects it
  • every other module that imports it
  • collection itself, even for a run that deselects all its tests

no_duplicate_dict_keys_test at 25s is paid by anyone who touches anything it
transitively reaches.

Acceptance criteria

  • Each of the four is measured to find what it does at import — a fixture build, a corpus parse, a client construction — with the specific line named
  • Expensive module-scope work moves into a fixture or a lazily-evaluated helper, so it is paid by the tests that need it
  • No guard module constructs a network client at import; the Ollama line above is the instance to start from
  • Re-measured after: full guard-set import time quoted before and after
  • A guard caps import time per test module, so the next one is caught when it lands rather than when it blocks a push

Not the fix

Raising the pre-push budget. #16182 is a real and separate defect — the budget
scales with changed-file count rather than with the selected test set — but a
64-second import is a cost worth removing regardless of what the budget is.

Related: #16182 (the budget) · #15928 (blocked by this) · #16183 (blocked by #15928)

Activity

  1. added this to the Backlog milestone on Sep 12, 2026
  2. mrveiss commented on Sep 12, 2026

    @mrveiss
    OwnerAuthor

    Stale-issue sweep (never auto-closed — Dev_new_gui wasn't GitHub's default branch at merge time). Verified against current origin/main; leaving open.

    4 of 5 criteria MET: the PR body names the specific expensive lines per module (module-scope _sweep(), files_naming_the_indexer(), discover_pricing_tables()/Ollama client init) with measured times (25.5s/24.5s/11.4s); lazy _CACHE-backed accessors now exist (no_duplicate_dict_keys_test.py:173-185, man_page_indexer_path_resolves_15853_test.py:170-176, model_pricing_tables_agree_15912_test.py:233-255); the Ollama/LocalLLM client construction moved inside the now-lazy path, no longer at module import; before/after measured at 67s → 5.9s for the full guard-set import.

    NOT MET: AC5 — no guard exists anywhere in repo_tests/ that caps or asserts per-module import time going forward; a grep for import-duration assertions finds only an unenforced comment referencing the 128s pre-push budget.

    Delivered by merged PR #16189.

  3. modified the milestones: Backlog, v0.14.0 on Sep 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions