Skip to content

flaky(test): test_system_startup_performance fails locally when HF model cache is warm but the network HEAD is blocked #16994

Open

Description

@mrveiss

Problem

autobot-backend/system_benchmarks_performance_test.py::TestSystemPerformanceBenchmarks::test_system_startup_performance fails deterministically on at least one dev machine with a populated ~/.cache/huggingface/hub (both models--openai--clip-vit-base-patch32 and models--Salesforce--blip2-opt-2.7b present):

AssertionError: online vs offline model load differs: [(False, False), (False, True)] (#16404)
assert (False, False) == (False, True)

With a warning immediately before it:

WARNING 'network access blocked for a warm-construction no-network check (#15055)' thrown while requesting HEAD https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/processor_config.json
WARNING Retrying in 1s [Retry 1/5].
ERROR Failed to load vision models: Can't load processor for 'openai/clip-vit-base-patch32'...

The test's own comment (#16404) says huggingface_hub should fall back to the on-disk cache on a connection error under HF_HUB_OFFLINE=1 + _blocked_sockets() — that fallback did not happen for one of the two models on this run, despite both being present in the local cache. Reproduced 3/3 runs (deterministic on this machine, not a one-off flake).

Not caused by, and not blocking, #16990 — confirmed unrelated: different test, no shared code path, and this whole file's markers (slow/integration/distributed/performance) are excluded from the standard ci.yml job that gates a normal PR (-m "not integration and not slow and not distributed and not performance"); only the release-promotion-specific marker-tests.yml runs them, where the real CI runner's HF cache state is presumably clean/consistent. Filed separately per no-debris — worth root-causing (retry-exhaustion timing vs. a real cache-fallback regression in the pinned huggingface_hub version) since it could affect other local pre-push runs, or CI's own runner if its cache ever gets into the same partial state.

Acceptance criteria

  • Root cause identified: retry/timeout race in the offline fallback, a version-specific huggingface_hub behavior change, or a genuinely inconsistent local cache directory.
  • Either the test made resilient to the fallback timing, or the underlying huggingface_hub offline-fallback behavior fixed/pinned.
  • Reproduction steps recorded here don't reproduce after the fix.

Activity

  1. added this to the v0.9.5 - CI and guards milestone on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingtesting

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions