[eval][cli][doc] feat: add resumable eval cache with cache management commands#78

Merged

JoyboyBrian merged 23 commits intomainfrom

brian/eval

Feb 23, 2026

Commits on Feb 21, 2026

eval improvement Phase 1: Add filelock and xxhash dependencies to pyproject.toml and uv.lock
JoyboyBrian
committed
eval enhancement phase 2: Enhance EvalRunner with batch processing capabilities and additional result attributes. Introduced 'messages' and 'row_index' to EvalRunResult, and implemented 'run_batch'…
JoyboyBrian
committed
eval enhancement phase 3: Implemented 'run_batch' method for concurrent execution and added 'messages' and 'row_index' attributes to EvalRunResult, improving evaluation efficiency.
JoyboyBrian
committed
eval improvement phase 4: Implement rubric cache migration and enhance cache management commands in CLI
JoyboyBrian
committed
eval enhancement phase 5: Introduce improved error handling in cache management commands and optimize cache migration process. This update refines the migration function for the rubric cache, enhan…
JoyboyBrian
committed
Enhance evaluation framework: Introduce retry mechanism for failed runs in EvalOrchestrator, improve cache management in CLI, and refine error handling in cache operations. This update also optimiz…
JoyboyBrian
committed
Enhance evaluation framework: Add dataset fingerprint verification to prevent cache usage with modified datasets, improve cache data handling in EvalCommand, and refine orchestration results to inc…
JoyboyBrian
committed
Enhance EvalCommand output: Add user tips for cache re-running and logging samples. This update improves user guidance by suggesting the use of the --fresh flag for fresh evaluations and the --log-…
JoyboyBrian
committed
Enhance module fingerprinting and progress tracking: Introduce support for MCP module paths in compute_module_fingerprint, update EvalCommand to handle MCP specifications, and ensure prior complete…
JoyboyBrian
committed
Refine EvalOrchestrator's retry mechanism: Update handling of completed runs to retain only successful ones when retrying failed evaluations. Enhance cache management by ensuring only successful ru…
JoyboyBrian
committed

Commits on Feb 22, 2026

Enhance LLM client and evaluation framework: Add properties for API key and base URL in ExternalLLMClient for better encapsulation. Update JSON writing to ensure proper character encoding. Include …
JoyboyBrian
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

[eval][cli][doc] feat: add resumable eval cache with cache management commands#78

[eval][cli][doc] feat: add resumable eval cache with cache management commands#78
JoyboyBrian merged 23 commits intomainfrom
brian/eval

Commits on Feb 21, 2026

eval improvement Phase 1: Add filelock and xxhash dependencies to pyproject.toml and uv.lock

eval enhancement phase 2: Enhance EvalRunner with batch processing capabilities and additional result attributes. Introduced 'messages' and 'row_index' to EvalRunResult, and implemented 'run_batch'…

eval enhancement phase 3: Implemented 'run_batch' method for concurrent execution and added 'messages' and 'row_index' attributes to EvalRunResult, improving evaluation efficiency.

eval improvement phase 4: Implement rubric cache migration and enhance cache management commands in CLI

eval enhancement phase 5: Introduce improved error handling in cache management commands and optimize cache migration process. This update refines the migration function for the rubric cache, enhan…

Enhance evaluation framework: Introduce retry mechanism for failed runs in EvalOrchestrator, improve cache management in CLI, and refine error handling in cache operations. This update also optimiz…

Enhance evaluation framework: Add dataset fingerprint verification to prevent cache usage with modified datasets, improve cache data handling in EvalCommand, and refine orchestration results to inc…

Enhance EvalCommand output: Add user tips for cache re-running and logging samples. This update improves user guidance by suggesting the use of the --fresh flag for fresh evaluations and the --log-…

Enhance module fingerprinting and progress tracking: Introduce support for MCP module paths in compute_module_fingerprint, update EvalCommand to handle MCP specifications, and ensure prior complete…

Refine EvalOrchestrator's retry mechanism: Update handling of completed runs to retain only successful ones when retrying failed evaluations. Enhance cache management by ensuring only successful ru…

Commits on Feb 22, 2026

Enhance LLM client and evaluation framework: Add properties for API key and base URL in ExternalLLMClient for better encapsulation. Update JSON writing to ensure proper character encoding. Include …

Commits on Feb 23, 2026

Refactor error handling in cache operations: Replace print statements with logger warnings for cache write failures, invalid timeout values, and backup issues. This enhances logging consistency and…

Refactor cache functions for improved clarity and consistency: Rename internal functions to public, update error handling to catch broader exceptions, and enhance logging for cache write failures. …

Refactor logging in session and cache management: Replace print statements with logger warnings for better error handling in rubric cache migration and directory checks. Update exception handling i…

Enhance eval functionality and caching: Update documentation to include new caching features for `osmosis eval`, such as automatic result caching, resuming interrupted evaluations, and additional c…

Enhance CLI cache management: Introduce new commands for listing and removing cached evaluations in `osmosis eval`. Update documentation to reflect these changes, including filtering options for mo…

format

fix pyright type checking issue

address pyright type checking issues

Improve error handling and logging in evaluation components: Enhance cache flushing by adding detailed warnings for cache read failures. Update CLI and runner error handling to include comments for…

Enable address reuse for local server socket: Add socket option to allow address reuse in `find_available_port` function, improving port availability handling.

address cubic review

fix