You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
flaky(test): test_system_startup_performance fails locally when HF model cache is warm but the network HEAD is blocked #16994
autobot-backend/system_benchmarks_performance_test.py::TestSystemPerformanceBenchmarks::test_system_startup_performance fails deterministically on at least one dev machine with a populated ~/.cache/huggingface/hub (both models--openai--clip-vit-base-patch32 and models--Salesforce--blip2-opt-2.7b present):
AssertionError: online vs offline model load differs: [(False, False), (False, True)] (#16404)
assert (False, False) == (False, True)
With a warning immediately before it:
WARNING 'network access blocked for a warm-construction no-network check (#15055)' thrown while requesting HEAD https://huggingface.co/openai/clip-vit-base-patch32/resolve/main/processor_config.json
WARNING Retrying in 1s [Retry 1/5].
ERROR Failed to load vision models: Can't load processor for 'openai/clip-vit-base-patch32'...
The test's own comment (#16404) says huggingface_hub should fall back to the on-disk cache on a connection error under HF_HUB_OFFLINE=1 + _blocked_sockets() — that fallback did not happen for one of the two models on this run, despite both being present in the local cache. Reproduced 3/3 runs (deterministic on this machine, not a one-off flake).
Not caused by, and not blocking, #16990 — confirmed unrelated: different test, no shared code path, and this whole file's markers (slow/integration/distributed/performance) are excluded from the standard ci.yml job that gates a normal PR (-m "not integration and not slow and not distributed and not performance"); only the release-promotion-specific marker-tests.yml runs them, where the real CI runner's HF cache state is presumably clean/consistent. Filed separately per no-debris — worth root-causing (retry-exhaustion timing vs. a real cache-fallback regression in the pinned huggingface_hub version) since it could affect other local pre-push runs, or CI's own runner if its cache ever gets into the same partial state.
Acceptance criteria
Root cause identified: retry/timeout race in the offline fallback, a version-specific huggingface_hub behavior change, or a genuinely inconsistent local cache directory.
Either the test made resilient to the fallback timing, or the underlying huggingface_hub offline-fallback behavior fixed/pinned.
Reproduction steps recorded here don't reproduce after the fix.
Activity