Skip to content

fix(reliability): per-search wall-clock watchdog (#151 item 5) - #154

Merged
fcostaoliveira merged 1 commit into
masterfrom
fix/search-watchdog-151-5
Jul 20, 2026
Merged

fcostaoliveira merged 1 commit into
masterfrom
fix/search-watchdog-151-5

Conversation

@fcostaoliveira

Copy link
Copy Markdown
Collaborator

What

Addresses item 5 of #151 (Vertex/Redis reliability): a single search/mixed point that stalls — e.g. a proxy or connection-pool exhaustion at high --parallel — could hang the whole sweep silently for ~an hour with no output, so you couldn't tell a stuck run from a slow one.

How

Opt-in --search-timeout <secs> watchdog:

  • run_with_search_watchdog runs the search on the current thread, so the timed measurement path and its fidelity are untouched. A monitor thread only watches the clock: it logs progress every 60s while a point is in flight, and if the point exceeds the timeout it prints a diagnostic naming the stuck point (engine/dataset mode[parallel=N]) and aborts the process (exit 3) instead of hanging.
  • Wraps the measured search/mixed call and both warm-up calls.
  • Default 0.0 disables it → behavior unchanged. NaN/inf are treated as disabled.

Tests

  • watchdog_disabled_runs_inline_and_returns_value — 0/negative/NaN/inf all run f inline and return its value verbatim.
  • watchdog_enabled_fast_completion_returns_value — monitor observes completion (tx drop) and returns without aborting.
  • search_timeout_parses — flag defaults to disabled and parses an explicit value.

Full suite green on cargo +stable (clippy -D warnings, fmt, 496+82 tests).

Refs #151 (item 5).

🤖 Generated with Claude Code

A single search/mixed point that stalls — e.g. a proxy/connection-pool
exhaustion at high `parallel` — could hang the whole sweep silently for
~an hour with no output. Add an opt-in `--search-timeout <secs>` watchdog:

- `run_with_search_watchdog` runs the search on the CURRENT thread, so the
  timed measurement path and its fidelity are untouched. A monitor thread
  only watches the clock: logs progress every 60s while a point is in
  flight, and if the point exceeds the timeout prints a diagnostic naming
  the stuck point (engine/dataset/mode/parallel) and aborts (exit 3).
- Wraps the measured search/mixed call plus both warm-up calls.
- Default 0.0 disables it → behavior unchanged. NaN/inf treated as disabled.

Tests: watchdog disabled runs inline & returns value verbatim; enabled +
fast completion returns without aborting; `--search-timeout` parses.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@fcostaoliveira
fcostaoliveira merged commit fc3c0a2 into master Jul 20, 2026
16 checks passed
@fcostaoliveira
fcostaoliveira deleted the fix/search-watchdog-151-5 branch July 20, 2026 12:21
@github-actions

Copy link
Copy Markdown

Docker Build Validation

Platforms tested:

  • linux/amd64 (built and tested)
  • linux/arm64 (build validated)

Tests performed:

  • --help command
  • --describe datasets command
  • Integration test (random-100 dataset + Redis)
  • Multi-platform build validation (amd64 + arm64)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant