Skip to content

feat: add detector benchmark script#60

Open
sanmaxdev wants to merge 1 commit into
KRLabsOrg:mainfrom
sanmaxdev:feat/detector-benchmark
Open

feat: add detector benchmark script#60
sanmaxdev wants to merge 1 commit into
KRLabsOrg:mainfrom
sanmaxdev:feat/detector-benchmark

Conversation

@sanmaxdev

Copy link
Copy Markdown
Contributor

Summary

  • Add a detector latency and throughput benchmark script for reusable local measurements.
  • Support built-in sample data, JSONL benchmark cases, warmup runs, repeated measurements, and JSON output.
  • Cover benchmark timing and input validation in the existing inference test suite.

Testing

  • python3 -m pytest tests/test_inference_pytest.py -v -k "not TestAnswerStartToken"
  • python3 -m ruff format --check --diff lettucedetect/ tests/
  • python3 -m ruff check lettucedetect/ tests/ --extend-exclude lettucedetect/integrations/
  • python3 -m ruff check scripts/benchmark_detectors.py tests/test_inference_pytest.py
  • python3 scripts/benchmark_detectors.py --help

Closes #58

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Latency / throughput micro-benchmark for the detectors

1 participant