Simplify custom prompt handling & re-organize

[asamal4]

Simplify custom prompt handling & re-organize

Summary by CodeRabbit

• New Features

  • Standardized answer-correctness evaluation prompt with clear scoring (0.0–1.0) and required reasoning output.
  • Simplified imports for custom metrics through a consolidated entry point.

• Refactor

  • Streamlined evaluation flow by directly formatting the prompt, improving consistency and reducing overhead.
  • Removed a legacy parameter class used for prompt construction; update any code that referenced it directly.

[coderabbitai]

Walkthrough

Adds a custom metrics package initializer exporting key symbols, introduces a shared answer-correctness prompt template, and refactors CustomMetrics to use this template directly, removing the EvaluationPromptParams class and its prompt-construction method.

Changes

Cohort / File(s)	Summary
Package exports for custom metrics src/lightspeed_evaluation/core/metrics/custom/__init__.py	New package initializer re-exporting CustomMetrics, ANSWER_CORRECTNESS_PROMPT, and evaluate_tool_calls; updates __all__.
CustomMetrics prompt refactor src/lightspeed_evaluation/core/metrics/custom/custom.py	Removes EvaluationPromptParams class and _create_evaluation_prompt(...); imports and formats ANSWER_CORRECTNESS_PROMPT directly for answer correctness evaluation.
Prompt template introduction src/lightspeed_evaluation/core/metrics/custom/prompts.py	Adds ANSWER_CORRECTNESS_PROMPT constant with specified scoring instructions and output format.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant E as Evaluator
  participant CM as CustomMetrics
  participant P as prompts.ANSWER_CORRECTNESS_PROMPT
  participant L as LLM/Scorer

  E->>CM: evaluate_answer_correctness(query, response, expected)
  Note right of CM: Prompt now built via template (no EvaluationPromptParams)
  CM->>P: Format template with {query, response, expected_response}
  P-->>CM: Prompt string
  CM->>L: Submit prompt
  L-->>CM: Score + Reason
  CM-->>E: Evaluation result

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

A rabbit taps the keys with cheer,
“Goodbye, old params—be light, be clear!”
A prompt now sings in one swift thread,
From template’s loom the scores are read.
Exports aligned, the paths are neat—
Thump-thump! This change hops quick and sweet. 🐇✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Title Check	✅ Passed	The title accurately captures the primary change of simplifying custom prompt handling and indicates that code structure has been reorganized, which aligns with the modifications to prompt generation and module exports in this pull request.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[asamal4]

@VladimirKadlec @tisnik PTAL

[VladimirKadlec]

LGTM

[tisnik]

LGTM