Repository navigation
Vertex benchmark reliability: 401 handling, token refresh, --engines-file no-op, Redis M×EFC index collision, search watchdog #151
Description
Issues found running the full Redis-Cloud-vs-Vertex GloVe/gist/dbpedia/deep-image benchmark end-to-end on GCP (loadgen n2-standard-64, Vertex 2×n2d-standard-32 private-gRPC PSC, Redis Cloud Pro QPF 8×). These are distinct from the fairness fixes in #148 (which are merged and validated — see the note at the bottom). Ordered by how badly they hurt a real run.
Must-fix — these blocked runs
1. Vertex stream-update upload treats HTTP 401 as fatal
The upload path retries 429/quota via Retry-After, but a transient 401 aborts the entire upload mid-stream, leaving a partially populated index that then returns silently-wrong recall. On the gist-960 upload (1M×960d) a 401 hit around batch ~600 and killed the run after indexSyncTime had already advanced (partial data).
Fix: on 401, re-fetch the access token and retry the batch (same treatment as 429). Workaround used: an external retry loop around the whole upload (stream-update upserts are idempotent by ID, so repeated runs fill the gaps).
engine/vertex.rs upload path (429 retry present, 401 not).
2. Access token is fetched once, not refreshed
access_token() uses VERTEX_ACCESS_TOKEN if set, else a one-shot gcloud auth print-access-token. For a long upload (deep-image-96 = 9.99M vectors, multi-hour with stream-update throttling) a 1-hour token expires mid-upload → cascading 401s (compounds #1).
Fix: use an auto-refreshing credential (google-auth / ADC token source that refreshes on expiry), or re-mint per batch / on 401.
3. --engines-file is a silent no-op
experiment.rs calls read_engine_configs() with no argument — it globs experiments/configurations/*.json and ignores --engines_file entirely, even though the README documents and uses it. Symptom: --engines-file my.json --engines my-config → "No engines match pattern" (it's listing built-in engine types, because the file was never read).
Fix: wire the flag through read_engine_configs(args.engines_file), or remove the flag + README references. Workaround: overwrite the bundled experiments/configurations/redis-single-node.json / vertex-single-node.json.
4. Redis M×EFC sweep silently collapses to a single index
configured_index_name() returns one fixed name (REDIS_INDEX_NAME or "idx") and create_index() runs FLUSHALL on every upload. So "upload all 12 M×EFC configs, then --skip-upload search" leaves only the last config's index alive — every search queries it → identical recall across all configs (observed a flat 0.917 for redis-m-16-ef-64 through redis-m-64-ef-512). Silent wrong data.
Fix (any of): incorporate the config name / M / EFC into the index name so collisions can't happen; emit a hard error/warning when two configs would target the same index; or document loudly that Redis must run upload+search per config (no --skip-upload), which is the only correct pattern given per-upload FLUSHALL.
Correctness / robustness
5. No per-search watchdog — a hung search stalls the whole sweep
One Redis parallel=256 closed-loop search hung indefinitely (~1 hr, zero log progress) and blocked the remaining sweep. Likely proxy/connection-pool exhaustion at 256 conns on a 16-vCPU node, but the harness should not stall.
Fix: wall-clock timeout per search point; on timeout, record a failure and continue.
6. --repetitions defaults to 3
In --search-duration closed-loop mode this triples runtime for an upward-biased max. Default to 1 in duration mode (or warn). (#148 review already flagged best-of-N as meaningless in open-loop.)
7. No auto-wait for Vertex indexSyncTime after upload
Queries are inaccurate until the stream-update sync clock advances past the upload; the harness relies on the operator to wait. Poll it before the search phase.
Nice-to-have
- 8. Result JSONs are huge (~255 MB for one dataset — per-query latency histograms). Streaming percentiles (t-digest/HDR) would cut memory + download.
- 9. Vertex
indexes create/deploy-indexare async (return before completion); any auto-provisioning must poll the operation.
What validated well
The #148 fairness fixes work in practice. With the prime+barrier, hoisted request construction, and the demoted parallel>cores saturation flag, Vertex reached its true 45,478 QPS @ 0.96 recall at parallel=256 — the earlier concurrency-64 runs capped it at ~24.6K, understating Vertex vs Google's own published 44,876 QPS. The fairness work materially corrected the result (in the competitor's favor, honestly), which is exactly what it was for.
Environment: GCP us-east1, tool @ master (8303f1d), 2026-07-20.
Activity
fix(#151): --engines-file no-op, Vertex upload 401 retry, repetitions…
fcostaoliveira commentedon Jul 20, 2026
Fixed the localized, validatable must-fixes in #152 (merged):
--engines-fileno-op —read_engine_configs()now reads the given file (ONLY that file) instead of always globbingexperiments/configurations/*.json; a missing/malformed file is a hard error. Verified end-to-end (--engines-file x.json --engines my-cfgnow runs the file's config) + unit test.Retry-Afterhandling), so a partial-index-then-wrong-recall can't happen. Together with the per-worker 45-min proactive token refresh already in the upload path, this also covers Update.redisearch hybrid benchmarks #2 for long uploads.--repetitionsin--search-durationmode — forced to 1 (best-of-N is meaningless in timed mode), warning if the user set >1.Remaining, each deserving its own PR:
"idx"and it needs Redis/Valkey/Dragonfly Docker validation, so it's split out rather than risk the well-tested Redis paths.indexSyncTimepost-upload wait, Add REDIS_KEEP_DOCUMENTS and --only-configure flag #8 streaming percentiles (t-digest/HDR). TOP CPU consumers on Billion Scale #9 async-op polling is already implemented in the Vertex engine.Thanks for the detailed report — and for confirming the #148 fairness work corrected Vertex's result upward (45,478 QPS @ 0.96).
fix(#151-4): per-config index+key namespacing so M×EFC sweeps don't c…
fcostaoliveira commentedon Jul 20, 2026
Item #4 (Redis/Valkey/Dragonfly M×EFC index collision) is now fixed in #153 (merged).
The root cause was a triple collision — fixed index name (
idx), bare doc keys (0,1,…), and an empty-PREFIXindex that covers all keys — plus destructive cleanup (redisDROPINDEX DDon that all-keys index; valkey/dragonflyFLUSHALL) that wiped the previous config on every upload.Fix: each config now gets its own index name
<base>:<config>+ doc-key prefix<config>:(FT.CREATE … PREFIX 1 "<config>:"), result keys are stripped back to ids on read, and cleanup is prefix-scoped (redis DD; valkey/dragonflySCAN+UNLINK, noFLUSHALL). So "upload all configs →--skip-uploadsearch each" now returns each config's own graph. Any remaining mismatch is a hard error (a startup name-collision guard + a per-searchFT.INFOexistence guard), never silent wrong data.Design was chosen and attack-hardened by a 15-agent adversarial pass; the diff was independently reviewed and live-validated on real Redis + Valkey + Dragonfly (coexistence returns distinct per-config data;
--skip-uploadwith no index fails loudly): redis 26/26, valkey 20/20, dragonfly 8/8.Behavior note (README updated):
*_INDEX_NAMEenv vars are now a base namespace (config always suffixed;*_INDEX_NAME_EXACT=1opts out); per-config prefixing stores N copies of otherwise-identical sweep docs (keyspace ×N — the isolation trade); indexes from a pre-fix binary are incompatible (re-upload).Remaining open items: #5 (per-search watchdog), #7 (Vertex indexSyncTime wait), #8 (streaming percentiles).
fix(reliability): per-search wall-clock watchdog (#151 item 5) (#154)
fix(vertex): wait for STREAM_UPDATE index to sync before search (#151 …
fix(results): HDR/quality digests replace raw per-query arrays (#151 …
fcostaoliveira commentedon Jul 20, 2026
All items in this issue are now addressed and merged to
master. Item → PR mapping:TOKEN_REFRESH_AFTER) reinforced by #152's 401 retry--engines-filesilent no-op → flag wired throughread_engine_configs--search-timeoutbounds a hung search point, logs progress, aborts with a diagnostic--repetitionsdefault 3 misapplied in duration mode → forced to 1 with a noteindexSyncTimeafter upload → pollindexStats.vectorsCountuntil synced before search--dump-raw-latenciesopt-in restores rawEach fix landed with unit + integration coverage on green CI; the Vertex-side changes (#155) and the Redis-family index work (#153) were additionally validated end-to-end on live GCP / Docker. Closing as complete.
Merge origin/master (#271) into #236, threading the derived key throu…
fix(mongodb): give every config its own collection and search index (#…