Repository navigation
fix(reliability): per-search wall-clock watchdog (#151 item 5) - #154
Merged
Merged
Conversation
A single search/mixed point that stalls — e.g. a proxy/connection-pool exhaustion at high `parallel` — could hang the whole sweep silently for ~an hour with no output. Add an opt-in `--search-timeout <secs>` watchdog: - `run_with_search_watchdog` runs the search on the CURRENT thread, so the timed measurement path and its fidelity are untouched. A monitor thread only watches the clock: logs progress every 60s while a point is in flight, and if the point exceeds the timeout prints a diagnostic naming the stuck point (engine/dataset/mode/parallel) and aborts (exit 3). - Wraps the measured search/mixed call plus both warm-up calls. - Default 0.0 disables it → behavior unchanged. NaN/inf treated as disabled. Tests: watchdog disabled runs inline & returns value verbatim; enabled + fast completion returns without aborting; `--search-timeout` parses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Docker Build ValidationPlatforms tested:
Tests performed:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Addresses item 5 of #151 (Vertex/Redis reliability): a single search/mixed point that stalls — e.g. a proxy or connection-pool exhaustion at high
--parallel— could hang the whole sweep silently for ~an hour with no output, so you couldn't tell a stuck run from a slow one.How
Opt-in
--search-timeout <secs>watchdog:run_with_search_watchdogruns the search on the current thread, so the timed measurement path and its fidelity are untouched. A monitor thread only watches the clock: it logs progress every 60s while a point is in flight, and if the point exceeds the timeout it prints a diagnostic naming the stuck point (engine/dataset mode[parallel=N]) and aborts the process (exit 3) instead of hanging.0.0disables it → behavior unchanged. NaN/inf are treated as disabled.Tests
watchdog_disabled_runs_inline_and_returns_value— 0/negative/NaN/inf all runfinline and return its value verbatim.watchdog_enabled_fast_completion_returns_value— monitor observes completion (tx drop) and returns without aborting.search_timeout_parses— flag defaults to disabled and parses an explicit value.Full suite green on
cargo +stable(clippy-D warnings, fmt, 496+82 tests).Refs #151 (item 5).
🤖 Generated with Claude Code