Conversation
|
Caution Review failedThe pull request is closed. WalkthroughThis PR adds src/draive/evaluators/utils.py (FORMAT_INSTRUCTION, tag-name constants, extract_evaluation_result) and converts ~17 evaluator modules to use FORMAT_INSTRUCTION and return extract_evaluation_result(await Stage.completion(...).execute()) instead of manual RESULT-tag parsing. In src/draive/evaluation/evaluator.py, Evaluator.lowest and Evaluator.highest signatures gain a concurrent_tasks parameter and now use haiway.concurrently (return_exceptions=False) instead of asyncio.gather; a new static Evaluator.average method is added to compute mean scores via concurrently. Evaluator.call now records an ObservabilityLevel.INFO metric. Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25–30 minutes Possibly related PRs
Pre-merge checks and finishing touches❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (1 passed)
📜 Recent review detailsConfiguration used: CodeRabbit UI Review profile: ASSERTIVE Plan: Pro 📒 Files selected for processing (20)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
src/draive/evaluators/tone_style.py (1)
9-44: Declare INSTRUCTION as FinalUse a module constant annotation for clarity and immutability.
-INSTRUCTION: str = f"""\ +from typing import Final +INSTRUCTION: Final[str] = f"""\src/draive/evaluators/coherence.py (1)
9-39: Mark INSTRUCTION as a final constant.Annotate INSTRUCTION with Final to convey immutability and intent.
-INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\Add import (top of file):
+from typing import Finalsrc/draive/evaluators/truthfulness.py (1)
9-40: Declare INSTRUCTION as Final.Same rationale as other evaluators; improves clarity and prevents accidental reassignment.
-INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\Add import:
+from typing import Finalsrc/draive/evaluators/readability.py (1)
9-40: Treat INSTRUCTION as an immutable constant.Use Final for the instruction template.
-INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\Add import:
+from typing import Finalsrc/draive/evaluators/similarity.py (1)
18-43: Make INSTRUCTION Final and consider full 5‑level scale for consistency.
- Final annotation matches other modules.
- Most evaluators use a 5‑level scale (poor/fair/good/excellent/perfect). Consider aligning to aid downstream averaging/comparisons.
-INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\ @@ <RATING> -Assign a similarity score using the exact name of one of the following values: -- "poor" is very low similarity; the content is completely unrelated in meaning. -- "good" is moderate similarity; the content shares some common themes or ideas. -- "perfect" is very high similarity; the content is very close in meaning\ - or conveys the same information. +Assign a similarity score using the exact name of one of the following values: +- "poor" is very low similarity; the content is completely unrelated in meaning. +- "fair" is low similarity; the content shares few overlapping elements. +- "good" is moderate similarity; the content shares some common themes or ideas. +- "excellent" is high similarity; the content is close in meaning with minor differences. +- "perfect" is very high similarity; the content is very close in meaning or conveys the same information. Use the "none" value for content that cannot be rated at all. </RATING>Add import:
+from typing import Final
♻️ Duplicate comments (10)
src/draive/evaluators/relevance.py (2)
9-40: Same INSTRUCTION constant nit as tone_styleMark as Final as suggested for tone_style.py.
63-78: Add logging/metrics and skip empty guidelinesApply the same small refactor as in tone_style.py (Lines 94–109).
src/draive/evaluators/coverage.py (2)
9-37: Make INSTRUCTION FinalSame as prior files.
60-75: Add logging/metrics and ignore blank guidelinesSame as tone_style suggestion.
src/draive/evaluators/factual_accuracy.py (2)
9-44: Declare INSTRUCTION as FinalAlign with other modules.
84-97: Add logging/metrics and handle blank guidelinesMirror tone_style refactor.
src/draive/evaluators/completeness.py (2)
9-42: Make INSTRUCTION FinalSame nit as others.
91-106: Add logging/metrics and skip empty guidelinesMirror tone_style improvement.
src/draive/evaluators/creativity.py (2)
9-45: Declare INSTRUCTION as FinalConsistent with other evaluators.
84-97: Add logging/metrics and ignore blank guidelinesSame as tone_style improvement.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (19)
src/draive/evaluation/evaluator.py(6 hunks)src/draive/evaluators/coherence.py(3 hunks)src/draive/evaluators/completeness.py(3 hunks)src/draive/evaluators/conciseness.py(3 hunks)src/draive/evaluators/consistency.py(3 hunks)src/draive/evaluators/coverage.py(4 hunks)src/draive/evaluators/creativity.py(4 hunks)src/draive/evaluators/expectations.py(4 hunks)src/draive/evaluators/factual_accuracy.py(4 hunks)src/draive/evaluators/fluency.py(4 hunks)src/draive/evaluators/groundedness.py(4 hunks)src/draive/evaluators/helpfulness.py(4 hunks)src/draive/evaluators/readability.py(4 hunks)src/draive/evaluators/relevance.py(4 hunks)src/draive/evaluators/safety.py(4 hunks)src/draive/evaluators/similarity.py(5 hunks)src/draive/evaluators/tone_style.py(4 hunks)src/draive/evaluators/truthfulness.py(4 hunks)src/draive/evaluators/utils.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Use Python 3.12+ features and syntax across the codebase
Format code exclusively with Ruff (make format); do not use other formatters
Skip module-level docstrings
Files:
src/draive/evaluators/tone_style.pysrc/draive/evaluators/coverage.pysrc/draive/evaluators/fluency.pysrc/draive/evaluators/coherence.pysrc/draive/evaluators/similarity.pysrc/draive/evaluators/factual_accuracy.pysrc/draive/evaluators/utils.pysrc/draive/evaluators/helpfulness.pysrc/draive/evaluation/evaluator.pysrc/draive/evaluators/creativity.pysrc/draive/evaluators/relevance.pysrc/draive/evaluators/consistency.pysrc/draive/evaluators/readability.pysrc/draive/evaluators/groundedness.pysrc/draive/evaluators/truthfulness.pysrc/draive/evaluators/completeness.pysrc/draive/evaluators/conciseness.pysrc/draive/evaluators/safety.pysrc/draive/evaluators/expectations.py
src/draive/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/draive/**/*.py: Import Haiway symbols directly (from haiway import State, ctx)
Use ctx.scope(...) to bind scoped Disposables and active State; avoid global state
Route all logs through ctx.log_debug/info/warn/error; do not use print
Use latest, most strict typing syntax (Python 3.12+), with strict typing only for public APIs
Avoid loose Any except at explicit third‑party boundaries
Prefer explicit attribute access with static types; avoid dynamic getattr except at narrow boundaries
Prefer Mapping/Sequence/Iterable in public types over dict/list/set
Use final where applicable; avoid inheritance and prefer composition
Use precise unions (|) and narrow with match/isinstance; avoid cast unless provably safe and localized
Model immutable data/config and facades with haiway.State; provide ergonomic classmethods like .of(...)
Avoid in-place mutation; use State.updated(...) or functional builders to produce new instances
Access active state via haiway.ctx inside async scopes (ctx.scope(...))
Use @statemethod for public state methods that dispatch on the active instance
Log around generation calls, tool dispatch, and provider requests/responses without leaking secrets; prefer structured/concise messages
Add metrics via ctx.record where applicable
All I/O is async; keep boundaries async and use ctx.spawn for detached tasks
Use structured concurrency and valid coroutine usage; rely on haiway/asyncio; avoid custom threading
Construct multimodal content with MultimodalContent.of(...) and compose blocks explicitly
Use ResourceContent/ResourceReference for media/data blobs
Wrap custom types/data within ArtifactContent; use hidden when needed
Add NumPy-style docstrings for public symbols with Parameters/Returns/Raises and rationale when non-obvious
Avoid docstrings on internal helpers; keep names self-explanatory
Keep docstrings high-quality; mkdocstrings pulls them into API reference
Never log secrets or full request bodies containing keys/tokens
Files:
src/draive/evaluators/tone_style.pysrc/draive/evaluators/coverage.pysrc/draive/evaluators/fluency.pysrc/draive/evaluators/coherence.pysrc/draive/evaluators/similarity.pysrc/draive/evaluators/factual_accuracy.pysrc/draive/evaluators/utils.pysrc/draive/evaluators/helpfulness.pysrc/draive/evaluation/evaluator.pysrc/draive/evaluators/creativity.pysrc/draive/evaluators/relevance.pysrc/draive/evaluators/consistency.pysrc/draive/evaluators/readability.pysrc/draive/evaluators/groundedness.pysrc/draive/evaluators/truthfulness.pysrc/draive/evaluators/completeness.pysrc/draive/evaluators/conciseness.pysrc/draive/evaluators/safety.pysrc/draive/evaluators/expectations.py
🧬 Code graph analysis (19)
src/draive/evaluators/tone_style.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
src/draive/evaluators/coverage.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/fluency.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/coherence.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/similarity.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/factual_accuracy.py (3)
src/draive/evaluation/score.py (1)
EvaluationScore(15-215)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/utils.py (2)
src/draive/evaluation/score.py (1)
EvaluationScore(15-215)src/draive/multimodal/content.py (2)
tag(405-425)tags(427-446)
src/draive/evaluators/helpfulness.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
src/draive/evaluation/evaluator.py (1)
src/draive/evaluation/value.py (1)
evaluation_score_value(32-98)
src/draive/evaluators/creativity.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
src/draive/evaluators/relevance.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/consistency.py (1)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)
src/draive/evaluators/readability.py (3)
src/draive/evaluation/score.py (2)
EvaluationScore(15-215)of(32-56)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/groundedness.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/truthfulness.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/completeness.py (4)
src/draive/evaluation/score.py (1)
EvaluationScore(15-215)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (3)
Stage(75-2042)completion(288-372)execute(1978-2042)
src/draive/evaluators/conciseness.py (2)
src/draive/evaluation/score.py (2)
EvaluationScore(15-215)of(32-56)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)
src/draive/evaluators/safety.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
src/draive/evaluators/expectations.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
🔇 Additional comments (19)
src/draive/evaluation/evaluator.py (2)
293-293: LGTM: Docstring clarifications improve accuracy.The updated docstrings correctly reflect that these methods compare
performancepercentage rather than rawscorevalues, which is consistent with the implementation at lines 324 and 369.Also applies to: 338-338
322-322: Consider exception handling strategy for composed evaluators.The explicit
return_exceptions=Falsemakes the behavior clear but is redundant since it's the default. More importantly, if any evaluator fails, the entire composition fails. Consider whether composed evaluators should be more resilient.For example, if one evaluator raises an exception, should the composition:
- Propagate the exception (current behavior)
- Log and skip the failed evaluator
- Return a fallback score for the failed evaluator
The current approach is consistent across
lowest,highest, andaverage, but may not be ideal for production use cases where partial results are valuable.Also applies to: 367-367
src/draive/evaluators/helpfulness.py (1)
1-2: LGTM: Consistent refactoring to use centralized result extraction.The changes properly delegate result parsing to the shared
extract_evaluation_resultutility and update the instruction format to useFORMAT_INSTRUCTIONplaceholder. This refactoring improves maintainability by centralizing the evaluation result parsing logic.Also applies to: 9-9, 42-42, 92-107
src/draive/evaluators/consistency.py (1)
1-2: LGTM: Consistent refactoring pattern applied.The refactoring follows the same pattern as other evaluators, properly using the centralized
extract_evaluation_resultutility andFORMAT_INSTRUCTIONplaceholder. The changes improve code consistency across evaluators.Also applies to: 9-9, 38-38, 62-77
src/draive/evaluators/fluency.py (1)
1-2: LGTM: Refactoring applied consistently.The changes maintain consistency with the refactoring pattern used across all evaluators, properly delegating to the shared utility for result extraction.
Also applies to: 9-9, 34-34, 50-63
src/draive/evaluators/expectations.py (1)
1-2: LGTM: Refactoring pattern consistently applied.The changes follow the established refactoring pattern, using the centralized extraction utility and format instruction placeholder.
Also applies to: 9-9, 33-33, 57-72
src/draive/evaluators/safety.py (1)
1-2: LGTM: Consistent refactoring to centralized utility.The safety evaluator has been updated to use the shared result extraction utility, maintaining consistency with the refactoring applied across all evaluators.
Also applies to: 9-9, 42-42, 82-95
src/draive/evaluators/groundedness.py (1)
1-2: LGTM: Refactoring applied consistently.The groundedness evaluator follows the same refactoring pattern as all other evaluators, using the centralized extraction utility and format instruction placeholder.
Also applies to: 9-9, 39-39, 63-78
src/draive/evaluators/conciseness.py (1)
1-2: LGTM: Refactoring completed consistently across all evaluators.The conciseness evaluator completes the consistent refactoring pattern applied across all evaluator files. The centralized result extraction utility improves maintainability and reduces code duplication.
Also applies to: 9-9, 40-40, 64-79
src/draive/evaluators/tone_style.py (1)
1-2: Centralized parsing + format placeholder — LGTMImports and delegation to utils are correct and reduce duplication.
src/draive/evaluators/relevance.py (1)
1-2: LGTMImports and extraction refactor look good.
src/draive/evaluators/coverage.py (1)
1-2: LGTMRefactor to utils is correct.
src/draive/evaluators/factual_accuracy.py (1)
1-2: LGTMImports and use of extract_evaluation_result are correct.
src/draive/evaluators/completeness.py (1)
1-2: LGTMRefactor and imports are correct.
src/draive/evaluators/creativity.py (1)
1-2: LGTMImports + centralized extraction look good.
src/draive/evaluators/coherence.py (1)
62-77: Centralized result extraction LGTM.Switching to extract_evaluation_result removes duplicate parsing and standardizes error paths. Good use of Stage.completion with formatted instructions.
src/draive/evaluators/truthfulness.py (1)
63-78: Result parsing refactor looks good.Delegating to extract_evaluation_result simplifies the evaluator and keeps behavior consistent across modules.
src/draive/evaluators/readability.py (1)
55-68: Good move to shared extractor.Cleaner and consistent with the new utilities.
src/draive/evaluators/similarity.py (1)
66-81: Unified extraction: approved.The change removes bespoke parsing and aligns with other evaluators.
KaQuMiQ
force-pushed
the
feature/evals
branch
2 times, most recently
from
3437009 to
c92b21e
Compare
There was a problem hiding this comment.
Actionable comments posted: 18
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/draive/evaluation/generator.py (1)
40-41: Tighten typing; avoid Any.Use the type parameter instead of
Anyforexample_pairs. As per coding guidelines.- example_pairs: list[tuple[str, Any]] = [(INPUT, example) for example in examples] + example_pairs: list[tuple[str, Parameters]] = [(INPUT, example) for example in examples]src/draive/evaluators/expectations.py (1)
24-29: Unquote rating names and standardize spellingAvoid quotes and consistently use “fulfillment” across bullets.
-Assign an expectation fulfillment score using exact name of one of the following values: -- "poor" is very low expectation fulfilment - the content misses most key points from the expectation. -- "fair" is low expectation fulfilment - the content includes some key points but omits several important ones. -- "good" is moderate expectation fulfilment - the content covers most key points but misses a few important details. -- "excellent" is high expectation fulfilment - the content includes nearly all key points with minor omissions. -- "perfect" is very high expectation fulfilment - the content comprehensively covers all key points from the expectations. +Assign an expectation fulfillment score using the exact name of one of the following values: +- poor — very low expectation fulfillment; the content misses most key points from the expectations. +- fair — low expectation fulfillment; the content includes some key points but omits several important ones. +- good — moderate expectation fulfillment; the content covers most key points but misses a few important details. +- excellent — high expectation fulfillment; the content includes nearly all key points with minor omissions. +- perfect — very high expectation fulfillment; the content comprehensively covers all key points from the expectations.
♻️ Duplicate comments (4)
src/draive/evaluation/evaluator.py (1)
376-420: Consider improving clarity of the average computation.The implementation is correct—at least one evaluator is always provided via the required positional parameter, ensuring
len(scores) > 0. However, previous review comments suggested making the divisor more explicit or inlining the computation to improve readability.Current approach using
Sequence[float]is good (addresses past feedback aboutMutableSequence).If you prefer more explicit code, consider this alternative:
async def evaluate( value: Value, ) -> EvaluatorResult: - scores: Sequence[float] = [ - result.score - for result in await gather( - evaluator(value), - *(evaluator(value) for evaluator in evaluators), - return_exceptions=False, - ) - ] - + results = await gather( + evaluator(value), + *(evaluator(value) for evaluator in evaluators), + return_exceptions=False, + ) + return EvaluatorResult( evaluator="average", - score=sum(scores) / len(scores), + score=sum(result.score for result in results) / len(results), threshold=evaluation_score_value(threshold), meta=META_EMPTY, )src/draive/evaluators/tone_style.py (1)
83-96: Consider handling whitespace-only guidelines.The current code at line 93 uses
if guidelines else "", which doesn't catch whitespace-only strings. This could result in empty<GUIDELINES></GUIDELINES>blocks being sent to the model.This concern was raised in a previous review comment on this same range.
Consider this refinement:
instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", + guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if (guidelines and guidelines.strip()) else "", ),src/draive/evaluators/utils.py (2)
17-23: Tighten RATING format: forbid punctuation as wellAdd “no punctuation” to reduce parser breaks caused by trailing periods/commas.
<REASONING>Concise, step-by-step justification that supports the rating.</REASONING> - <RATING>Selected rating value with exactly one of the allowed rating names (no quotes or extra text)</RATING> + <RATING>Selected rating value with exactly one of the allowed rating names (no quotes, no punctuation, no extra text)</RATING>
43-46: Harden parsing: normalize rating and omit empty metaStrip quotes and trailing dot, lowercase, and only include comment when present. Prevents failures when models output
"good".etc.- return EvaluationScore.of( - cast(EvaluationScoreValue, rating.strip().lower()), - meta={"comment": reasoning}, - ) + normalized = ( + rating.strip().strip('"').strip("'").rstrip(".").lower() + if rating is not None + else "" + ) + return EvaluationScore.of( + cast(EvaluationScoreValue, normalized), + meta=({"comment": reasoning} if reasoning else None), + )
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (20)
src/draive/evaluation/evaluator.py(6 hunks)src/draive/evaluation/generator.py(1 hunks)src/draive/evaluators/coherence.py(2 hunks)src/draive/evaluators/completeness.py(2 hunks)src/draive/evaluators/conciseness.py(2 hunks)src/draive/evaluators/consistency.py(3 hunks)src/draive/evaluators/coverage.py(3 hunks)src/draive/evaluators/creativity.py(2 hunks)src/draive/evaluators/expectations.py(3 hunks)src/draive/evaluators/factual_accuracy.py(2 hunks)src/draive/evaluators/fluency.py(2 hunks)src/draive/evaluators/groundedness.py(3 hunks)src/draive/evaluators/helpfulness.py(2 hunks)src/draive/evaluators/readability.py(3 hunks)src/draive/evaluators/relevance.py(2 hunks)src/draive/evaluators/safety.py(2 hunks)src/draive/evaluators/similarity.py(3 hunks)src/draive/evaluators/tone_style.py(2 hunks)src/draive/evaluators/truthfulness.py(3 hunks)src/draive/evaluators/utils.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Use Python 3.12+ features and syntax across the codebase
Format code exclusively with Ruff (make format); do not use other formatters
Skip module-level docstrings
Files:
src/draive/evaluators/factual_accuracy.pysrc/draive/evaluators/relevance.pysrc/draive/evaluators/similarity.pysrc/draive/evaluators/truthfulness.pysrc/draive/evaluators/helpfulness.pysrc/draive/evaluators/creativity.pysrc/draive/evaluation/generator.pysrc/draive/evaluators/fluency.pysrc/draive/evaluators/consistency.pysrc/draive/evaluators/completeness.pysrc/draive/evaluators/coherence.pysrc/draive/evaluators/coverage.pysrc/draive/evaluators/safety.pysrc/draive/evaluators/utils.pysrc/draive/evaluators/conciseness.pysrc/draive/evaluation/evaluator.pysrc/draive/evaluators/groundedness.pysrc/draive/evaluators/readability.pysrc/draive/evaluators/expectations.pysrc/draive/evaluators/tone_style.py
src/draive/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/draive/**/*.py: Import Haiway symbols directly (from haiway import State, ctx)
Use ctx.scope(...) to bind scoped Disposables and active State; avoid global state
Route all logs through ctx.log_debug/info/warn/error; do not use print
Use latest, most strict typing syntax (Python 3.12+), with strict typing only for public APIs
Avoid loose Any except at explicit third‑party boundaries
Prefer explicit attribute access with static types; avoid dynamic getattr except at narrow boundaries
Prefer Mapping/Sequence/Iterable in public types over dict/list/set
Use final where applicable; avoid inheritance and prefer composition
Use precise unions (|) and narrow with match/isinstance; avoid cast unless provably safe and localized
Model immutable data/config and facades with haiway.State; provide ergonomic classmethods like .of(...)
Avoid in-place mutation; use State.updated(...) or functional builders to produce new instances
Access active state via haiway.ctx inside async scopes (ctx.scope(...))
Use @statemethod for public state methods that dispatch on the active instance
Log around generation calls, tool dispatch, and provider requests/responses without leaking secrets; prefer structured/concise messages
Add metrics via ctx.record where applicable
All I/O is async; keep boundaries async and use ctx.spawn for detached tasks
Use structured concurrency and valid coroutine usage; rely on haiway/asyncio; avoid custom threading
Construct multimodal content with MultimodalContent.of(...) and compose blocks explicitly
Use ResourceContent/ResourceReference for media/data blobs
Wrap custom types/data within ArtifactContent; use hidden when needed
Add NumPy-style docstrings for public symbols with Parameters/Returns/Raises and rationale when non-obvious
Avoid docstrings on internal helpers; keep names self-explanatory
Keep docstrings high-quality; mkdocstrings pulls them into API reference
Never log secrets or full request bodies containing keys/tokens
Files:
src/draive/evaluators/factual_accuracy.pysrc/draive/evaluators/relevance.pysrc/draive/evaluators/similarity.pysrc/draive/evaluators/truthfulness.pysrc/draive/evaluators/helpfulness.pysrc/draive/evaluators/creativity.pysrc/draive/evaluation/generator.pysrc/draive/evaluators/fluency.pysrc/draive/evaluators/consistency.pysrc/draive/evaluators/completeness.pysrc/draive/evaluators/coherence.pysrc/draive/evaluators/coverage.pysrc/draive/evaluators/safety.pysrc/draive/evaluators/utils.pysrc/draive/evaluators/conciseness.pysrc/draive/evaluation/evaluator.pysrc/draive/evaluators/groundedness.pysrc/draive/evaluators/readability.pysrc/draive/evaluators/expectations.pysrc/draive/evaluators/tone_style.py
🧬 Code graph analysis (19)
src/draive/evaluators/factual_accuracy.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/relevance.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/similarity.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/stages/stage.py (3)
Stage(75-2042)completion(288-372)execute(1978-2042)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/truthfulness.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/helpfulness.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/creativity.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/fluency.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/consistency.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (3)
Stage(75-2042)completion(288-372)execute(1978-2042)
src/draive/evaluators/completeness.py (3)
src/draive/evaluation/score.py (1)
EvaluationScore(15-215)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
src/draive/evaluators/coherence.py (4)
src/draive/evaluation/score.py (2)
EvaluationScore(15-215)of(32-56)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/coverage.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/safety.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/utils.py (2)
src/draive/evaluation/score.py (1)
EvaluationScore(15-215)src/draive/multimodal/content.py (2)
tag(405-425)tags(427-446)
src/draive/evaluators/conciseness.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluation/evaluator.py (1)
src/draive/evaluation/value.py (1)
evaluation_score_value(32-98)
src/draive/evaluators/groundedness.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/readability.py (4)
src/draive/evaluation/score.py (2)
EvaluationScore(15-215)of(32-56)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/expectations.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/tone_style.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
🔇 Additional comments (21)
src/draive/evaluation/generator.py (1)
47-49: Behavior change: empty-string guidelines now omitted.Previously an empty string produced an empty block; now it’s omitted. Confirm this is intended for downstream evaluators/templates.
src/draive/evaluators/helpfulness.py (1)
82-95: Good centralization via extract_evaluation_result.Removes duplicated parsing; consistent with utils. LGTM.
src/draive/evaluators/coverage.py (1)
57-70: Good refactor to shared extractor.Simpler, uniform error handling. LGTM.
src/draive/evaluators/fluency.py (1)
46-57: Refactor looks good.Shared result extraction and streamlined Stage usage. LGTM.
src/draive/evaluators/relevance.py (1)
57-70: Refactor approved.Unified parsing via utils; consistent with other evaluators.
src/draive/evaluators/groundedness.py (1)
1-69: LGTM! Clean refactoring to centralized extraction utility.The migration to
extract_evaluation_resultandFORMAT_INSTRUCTIONis well-executed. The f-string INSTRUCTION with{{guidelines}}placeholder correctly escapes to single braces for.format(), and the guidelines formatting at line 66 properly includes the GUIDELINES block only when non-empty.src/draive/evaluators/safety.py (1)
1-83: LGTM! Consistent refactoring.The changes follow the same clean pattern as other evaluators, correctly delegating result parsing to the shared utility.
src/draive/evaluators/factual_accuracy.py (1)
1-84: LGTM! Refactoring is consistent.The changes correctly apply the centralized extraction pattern.
src/draive/evaluators/coherence.py (1)
1-71: LGTM! Clean migration to shared utilities.The refactoring is correct and maintains the evaluator's behavior.
src/draive/evaluators/truthfulness.py (1)
1-69: LGTM! Consistent with refactoring pattern.The changes correctly apply the centralized extraction utility.
src/draive/evaluation/evaluator.py (2)
322-322: LGTM! Explicit return_exceptions=False improves clarity.While
Falseis the default forgather(), making it explicit documents the intent that exceptions should propagate rather than being collected.
367-367: LGTM! Explicit return_exceptions=False improves clarity.Consistent with the
lowestevaluator and makes exception handling behavior clear.src/draive/evaluators/creativity.py (1)
1-83: LGTM! Refactoring is consistent and correct.The changes properly apply the centralized extraction pattern.
src/draive/evaluators/readability.py (1)
50-61: LGTM on centralized extractionGood switch to extract_evaluation_result and shared FORMAT usage.
src/draive/evaluators/completeness.py (1)
82-95: LGTM on shared extractor usageCentralization reduces duplication and failure modes.
src/draive/evaluators/conciseness.py (1)
57-70: LGTM on refactor to shared extractorClean and consistent with the new utility.
src/draive/evaluators/similarity.py (1)
63-76: LGTM on centralized extractionLooks correct and consistent with other evaluators.
src/draive/evaluators/expectations.py (1)
56-69: LGTM on shared extractor integrationClean replacement of ad-hoc parsing.
src/draive/evaluators/consistency.py (3)
1-2: LGTM: Clean centralization of evaluator utilities.The new imports correctly reference the centralized format instruction and result extraction utility, aligning with the PR's objective to eliminate inline result parsing.
9-34: LGTM: F-string pattern is correct but worth noting.The INSTRUCTION uses an f-string that embeds
FORMAT_INSTRUCTIONat module load time (line 33), while{{guidelines}}(line 22) is preserved with double braces for the runtime.format()call (line 66). This mixed-evaluation pattern is intentional and works correctly, but maintainers should be aware that the template has two evaluation phases.
57-70: LGTM: Clean refactoring with proper delegation.The evaluation flow is well-structured:
Stage.completion().execute()correctly returnsMultimodalContentextract_evaluation_result()centralizes tag extraction and error handling- Guidelines are conditionally formatted with proper XML wrapping
- Async/await usage is correct
The refactoring eliminates inline parsing logic while maintaining proper error handling through the utility function.
KaQuMiQ
force-pushed
the
feature/evals
branch
from
c92b21e to
29fe15e
Compare
There was a problem hiding this comment.
Actionable comments posted: 14
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (5)
src/draive/evaluation/evaluator.py (2)
311-327: lowest(): placeholder baseline can return an invalid synthetic result
Using a synthetic "lowest" result with score=1.0 and threshold=0 sets baseline performance to 100. If all real results have performance > 100 (e.g., high scores with low thresholds), no candidate will be <= 100 and the function returns the synthetic placeholder. Replace with computing results first and selecting min by performance.Apply:
async def evaluate( value: Value, ) -> EvaluatorResult: - # Placeholder for the lowest result - lowest: EvaluatorResult = EvaluatorResult( - evaluator="lowest", - score=1.0, - threshold=0, - meta=META_EMPTY, - ) - - for result in await gather( - evaluator(value), - *(evaluator(value) for evaluator in evaluators), - return_exceptions=False, - ): - if result.performance <= lowest.performance: - lowest = result - - return lowest + results = await gather( + evaluator(value), + *(evaluator(value) for evaluator in evaluators), + return_exceptions=False, + ) + return min(results, key=lambda r: r.performance)Also applies to: 319-323, 324-327
364-373: highest(): optional symmetry refactor
Works as-is. For symmetry with lowest(), consider selecting via max(...) after a single await gather to reduce branching.Apply:
- for result in await gather( - evaluator(value), - *(evaluator(value) for evaluator in evaluators), - return_exceptions=False, - ): - if result.performance >= highest.performance: - highest = result - - return highest + results = await gather( + evaluator(value), + *(evaluator(value) for evaluator in evaluators), + return_exceptions=False, + ) + return max(results, key=lambda r: r.performance)Also applies to: 365-368
src/draive/evaluators/truthfulness.py (1)
24-28: Remove quotes from rating namesUnquote rating examples to align with FORMAT_INSTRUCTION and prevent parsing issues.
Apply this diff:
Assign a truthfulness (factual accuracy) score using exact name of one of the following values: -- "poor" is very low truthfulness, the content contains many factual inaccuracies or distortions. -- "fair" is low truthfulness, the content has some accurate information but also significant factual errors or misrepresentations. -- "good" is moderate truthfulness, the content is somewhat accurate but with noticeable factual inaccuracies or distortions. -- "excellent" is high truthfulness, the content is mostly accurate with minimal factual errors or misrepresentations. -- "perfect" is very high truthfulness, the content is fully accurate, with no factual errors or misrepresentations. +- poor is very low truthfulness, the content contains many factual inaccuracies or distortions. +- fair is low truthfulness, the content has some accurate information but also significant factual errors or misrepresentations. +- good is moderate truthfulness, the content is somewhat accurate but with noticeable factual inaccuracies or distortions. +- excellent is high truthfulness, the content is mostly accurate with minimal factual errors or misrepresentations. +- perfect is very high truthfulness, the content is fully accurate, with no factual errors or misrepresentations.src/draive/evaluators/fluency.py (1)
36-41: Add a NumPy‑style docstring to the public evaluatorPublic APIs should have high‑quality docstrings (Parameters/Returns/Raises).
Example:
async def fluency_evaluator(... ) -> EvaluationScore: """ Evaluate fluency. Parameters ---------- evaluated : Multimodal guidelines : str | None Returns ------- EvaluationScore """src/draive/evaluators/consistency.py (1)
9-34: MakeINSTRUCTIONFinal and unquote rating tokensImmutability + parse‑friendly tokens.
+from typing import Final @@ -INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\ @@ -Assign a consistency score using exact name of one of the following values: -- "poor" is very low consistency, the content contains multiple hallucinated facts or significant misalignments with the reference content. -- "fair" is low consistency, the content has several instances of information not supported by the reference content. -- "good" is moderate consistency, the content is mostly consistent but contains a few unsupported statements. -- "excellent" is high consistency, the content is largely consistent with minor discrepancies. -- "perfect" is very high consistency, the content is fully consistent with the reference content, containing only supported information. +Assign a consistency score using the exact name of one of the following values: +- poor — very low consistency; multiple hallucinated facts or significant misalignments with the reference. +- fair — low consistency; several instances of information not supported by the reference. +- good — moderate consistency; mostly consistent but with a few unsupported statements. +- excellent — high consistency; largely consistent with minor discrepancies. +- perfect — very high consistency; fully consistent with the reference, containing only supported information. -Use the "none" value for content that cannot be rated at all. +Use the none value for content that cannot be rated at all.
♻️ Duplicate comments (19)
src/draive/evaluation/evaluator.py (2)
3-3: Import of Sequence looks good; resolves earlier nit.
Switching to Sequence is appropriate and matches immutability of the collected scores.
401-419: average(): implementation OK; minor clarity improvement optional
Good use of Sequence and explicit average. You can avoid the intermediate list and compute over the gathered results directly.Apply:
- scores: Sequence[float] = [ - result.score - for result in await gather( - evaluator(value), - *(evaluator(value) for evaluator in evaluators), - return_exceptions=False, - ) - ] - - return EvaluatorResult( - evaluator="average", - score=sum(scores) / len(scores), + results = await gather( + evaluator(value), + *(evaluator(value) for evaluator in evaluators), + return_exceptions=False, + ) + return EvaluatorResult( + evaluator="average", + score=sum(r.score for r in results) / len(results), threshold=evaluation_score_value(threshold), meta=META_EMPTY, )Also applies to: 404-416
src/draive/evaluation/generator.py (1)
46-50: Escape braces in guidelines before .format and skip blank-only values
Unescaped “{”/“}” in guidelines will break/poison str.format. Also skip empty/whitespace-only guidelines.Apply:
- instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" - if guidelines - else "" - ), + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if (guidelines and guidelines.strip()) + else "" + ), + ),Also applies to: 47-49
src/draive/evaluators/helpfulness.py (1)
91-93: Template safety: escape braces in guidelines and ignore blank values
Passing raw guidelines into INSTRUCTION.format risks KeyError/format injection when guidelines contains “{”/“}”. Escape braces and skip whitespace-only input.Apply:
- instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", - ), + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if (guidelines and guidelines.strip()) + else "" + ), + ),src/draive/evaluators/relevance.py (1)
66-68: Escape braces in guidelines and ignore blanks
Prevent str.format collisions/injection and avoid empty GUIDELINES output.Apply:
- instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", - ), + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if (guidelines and guidelines.strip()) + else "" + ), + ),src/draive/evaluators/tone_style.py (1)
92-95: Harden formatting (escape braces) and skip empty guidelines; optional logging/metrics
- Escape “{”/“}” in guidelines before INSTRUCTION.format.
- Skip whitespace-only guidelines.
- Optional: log/record around the stage call per project guidance. As per coding guidelines.
Apply:
- return extract_evaluation_result( - await Stage.completion( + result = await Stage.completion( MultimodalContent.of( @@ - instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", - ), - ).execute() - ) + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if (guidelines and guidelines.strip()) + else "" + ), + ), + ).execute() + ) + # from haiway import ctx # if not already imported at top-level + # ctx.log_debug("evaluator.tone_style.completed") # optional + # ctx.record("evaluator.tone_style.value", 1) # optional placeholder metric + return extract_evaluation_result(result)src/draive/evaluators/coverage.py (1)
66-68: Escape braces in guidelines and ignore blank-only values
Avoid str.format injection and empty GUIDELINES output.Apply:
- instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", - ), + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if (guidelines and guidelines.strip()) + else "" + ), + ),src/draive/evaluators/utils.py (2)
8-12: Export REASONING_TAG_NAME for API symmetryThe
__all__tuple exportsRATING_TAG_NAMEbut omitsREASONING_TAG_NAME, creating an asymmetric public API.Apply this diff:
__all__ = ( "FORMAT_INSTRUCTION", + "REASONING_TAG_NAME", "RATING_TAG_NAME", "extract_evaluation_result", )
26-46: Harden result parsing: strip quotes and handle None metaThe current parsing doesn't strip surrounding quotes from ratings (e.g.,
"poor"→poor), and always includes"comment"in meta even when reasoning isNone. This can cause parsing errors when models emit quoted ratings despite FORMAT_INSTRUCTION guidance.Apply this diff:
def extract_evaluation_result( content: MultimodalContent, /, ) -> EvaluationScore: reasoning: str | None = None rating: str | None = None for tag in content.tags(): name: str = tag.name.lower() if name == RATING_TAG_NAME: rating = tag.content.to_str() elif name == REASONING_TAG_NAME: reasoning = tag.content.to_str() if not rating: raise ValueError(f"Invalid evaluator result - missing rating:\n{content}") + # Strip whitespace and surrounding quotes/punctuation + normalized = rating.strip().strip('"').strip("'").rstrip(".").lower() + if not normalized: + raise ValueError(f"Invalid evaluator result - empty rating after normalization:\n{content}") + return EvaluationScore.of( - cast(EvaluationScoreValue, rating.strip().lower()), - meta={"comment": reasoning}, + cast(EvaluationScoreValue, normalized), + meta=({"comment": reasoning} if reasoning else None), )src/draive/evaluators/completeness.py (2)
9-9: Mark INSTRUCTION as FinalAdd
Finalannotation to prevent reassignment.Apply this diff:
+from typing import Final + from draive.evaluation import EvaluationScore, evaluator-INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\
24-28: Remove quotes from rating namesUnquote rating examples to align with FORMAT_INSTRUCTION.
Apply this diff:
Assign a completeness score using exact name of one of the following values: -- "poor" is very low completeness, the content addresses very few aspects of the user's query, leaving most questions unanswered. -- "fair" is low completeness, the content addresses some aspects of the user's query but leaves several important parts unanswered or incomplete. -- "good" is moderate completeness, the content addresses most aspects of the user's query but may miss some details or minor components. -- "excellent" is high completeness, the content addresses nearly all aspects of the user's query with only minor gaps or omissions. -- "perfect" is very high completeness, the content fully and comprehensively addresses all aspects of the user's query without any significant omissions. +- poor is very low completeness, the content addresses very few aspects of the user's query, leaving most questions unanswered. +- fair is low completeness, the content addresses some aspects of the user's query but leaves several important parts unanswered or incomplete. +- good is moderate completeness, the content addresses most aspects of the user's query but may miss some details or minor components. +- excellent is high completeness, the content addresses nearly all aspects of the user's query with only minor gaps or omissions. +- perfect is very high completeness, the content fully and comprehensively addresses all aspects of the user's query without any significant omissions.src/draive/evaluators/readability.py (2)
9-9: Mark INSTRUCTION as FinalAdd
Finalannotation for immutability.Apply this diff:
+from typing import Final + from draive.evaluation import EvaluationScore, evaluator-INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\
26-29: Remove quotes from rating namesUnquote rating examples to prevent parsing errors.
Apply this diff:
Assign a readability score using exact name of one of the following values: -- "poor" is very low readability, the content is extremely difficult to understand, with complex language and convoluted structure. -- "fair" is low readability, the content is challenging to read, with frequent use of complex sentences, unclear language or irrelevant parts. -- "good" is moderate readability, the content is somewhat clear but has some areas that are difficult to understand. -- "excellent" is high readability, the content is mostly clear and easy to read, with minor instances of complexity. +- poor is very low readability, the content is extremely difficult to understand, with complex language and convoluted structure. +- fair is low readability, the content is challenging to read, with frequent use of complex sentences, unclear language or irrelevant parts. +- good is moderate readability, the content is somewhat clear but has some areas that are difficult to understand. +- excellent is high readability, the content is mostly clear and easy to read, with minor instances of complexity.src/draive/evaluators/expectations.py (2)
9-33: UseFinal, unquote ratings, and standardize “fulfillment” spellingKeep constant immutable, avoid quoted outputs, and unify US spelling across the file.
+from typing import Final @@ -INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\ @@ -Assign an expectation fulfillment score using exact name of one of the following values: -- "poor" is very low expectation fulfilment - the content misses most key points from the expectation. -- "fair" is low expectation fulfilment - the content includes some key points but omits several important ones. -- "good" is moderate expectation fulfilment - the content covers most key points but misses a few important details. -- "excellent" is high expectation fulfilment - the content includes nearly all key points with minor omissions. -- "perfect" is very high expectation fulfilment - the content comprehensively covers all key points from the expectations. -Use the "none" value for content that cannot be rated at all. +Assign an expectation fulfillment score using the exact name of one of the following values: +- poor — very low expectation fulfillment; misses most key points from the expectations. +- fair — low expectation fulfillment; includes some key points but omits several important ones. +- good — moderate expectation fulfillment; covers most key points but misses a few important details. +- excellent — high expectation fulfillment; includes nearly all key points with minor omissions. +- perfect — very high expectation fulfillment; comprehensively covers all key points from the expectations. +Use the none value for content that cannot be rated at all.
65-67: Escape{/}inguidelinesbefore formattingSame issue as other modules; escape braces first.
- instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", - ), + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if guidelines else "" + ), + ),src/draive/evaluators/fluency.py (1)
55-57: Escape braces inguidelinesbeforeINSTRUCTION.formatUnescaped
{/}inguidelinescan break formatting or enable template injection. Escape first.- instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", - ), + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if guidelines else "" + ), + ),src/draive/evaluators/conciseness.py (2)
9-34: UseFinaland unquote rating tokensKeep constant immutable and avoid quoted outputs.
+from typing import Final @@ -INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\ @@ -Assign a conciseness score using exact name of one of the following values: -- "poor" is very low conciseness, the content is excessively verbose with much irrelevant information. -- "fair" is low conciseness, the content contains unnecessary details and some irrelevant information. -- "good" is moderate conciseness, the content is somewhat concise but could be more focused. -- "excellent" is high conciseness, the content is mostly concise with minimal unnecessary information. -- "perfect" is very high conciseness, the content is highly concise, containing only essential information. +Assign a conciseness score using the exact name of one of the following values: +- poor — very low conciseness; excessively verbose with much irrelevant information. +- fair — low conciseness; contains unnecessary details and some irrelevant information. +- good — moderate conciseness; somewhat concise but could be more focused. +- excellent — high conciseness; mostly concise with minimal unnecessary information. +- perfect — very high conciseness; highly concise, containing only essential information. -Use the "none" value for content that cannot be rated at all. +Use the none value for content that cannot be rated at all.
66-68: Escape{/}inguidelinesSame format‑collision risk as other evaluators.
- instructions=INSTRUCTION.format( - guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", - ), + instructions=INSTRUCTION.format( + guidelines=( + f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n" + if guidelines else "" + ), + ),src/draive/evaluators/similarity.py (1)
18-40: MakeINSTRUCTIONFinal and expand to 5‑level unquoted scale
- Use
Final[str]for immutability.- Align with standard 5 levels (poor, fair, good, excellent, perfect) and remove quotes.
+from typing import Final @@ -INSTRUCTION: str = f"""\ +INSTRUCTION: Final[str] = f"""\ @@ -Assign a similarity score using the exact name of one of the following values: -- "poor" is very low similarity; the content is completely unrelated in meaning. -- "good" is moderate similarity; the content shares some common themes or ideas. -- "perfect" is very high similarity; the content is very close in meaning or conveys the same information. +Assign a similarity score using the exact name of one of the following values: +- poor — very low similarity; the content is completely unrelated in meaning. +- fair — low similarity; limited overlap in themes or ideas. +- good — moderate similarity; shares some common themes or ideas. +- excellent — high similarity; closely aligned with the reference with minor differences. +- perfect — very high similarity; conveys essentially the same information. -Use the "none" value for content that cannot be rated at all. +Use the none value for content that cannot be rated at all.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro
📒 Files selected for processing (20)
src/draive/evaluation/evaluator.py(6 hunks)src/draive/evaluation/generator.py(1 hunks)src/draive/evaluators/coherence.py(2 hunks)src/draive/evaluators/completeness.py(2 hunks)src/draive/evaluators/conciseness.py(2 hunks)src/draive/evaluators/consistency.py(3 hunks)src/draive/evaluators/coverage.py(3 hunks)src/draive/evaluators/creativity.py(2 hunks)src/draive/evaluators/expectations.py(3 hunks)src/draive/evaluators/factual_accuracy.py(2 hunks)src/draive/evaluators/fluency.py(2 hunks)src/draive/evaluators/groundedness.py(3 hunks)src/draive/evaluators/helpfulness.py(2 hunks)src/draive/evaluators/readability.py(3 hunks)src/draive/evaluators/relevance.py(2 hunks)src/draive/evaluators/safety.py(2 hunks)src/draive/evaluators/similarity.py(3 hunks)src/draive/evaluators/tone_style.py(2 hunks)src/draive/evaluators/truthfulness.py(3 hunks)src/draive/evaluators/utils.py(1 hunks)
🧰 Additional context used
📓 Path-based instructions (2)
**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
**/*.py: Use Python 3.12+ features and syntax across the codebase
Format code exclusively with Ruff (make format); do not use other formatters
Skip module-level docstrings
Files:
src/draive/evaluators/groundedness.pysrc/draive/evaluators/consistency.pysrc/draive/evaluators/fluency.pysrc/draive/evaluators/expectations.pysrc/draive/evaluators/tone_style.pysrc/draive/evaluators/truthfulness.pysrc/draive/evaluators/readability.pysrc/draive/evaluators/utils.pysrc/draive/evaluators/conciseness.pysrc/draive/evaluators/factual_accuracy.pysrc/draive/evaluators/completeness.pysrc/draive/evaluators/similarity.pysrc/draive/evaluators/safety.pysrc/draive/evaluators/helpfulness.pysrc/draive/evaluators/relevance.pysrc/draive/evaluation/generator.pysrc/draive/evaluation/evaluator.pysrc/draive/evaluators/creativity.pysrc/draive/evaluators/coverage.pysrc/draive/evaluators/coherence.py
src/draive/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
src/draive/**/*.py: Import Haiway symbols directly (from haiway import State, ctx)
Use ctx.scope(...) to bind scoped Disposables and active State; avoid global state
Route all logs through ctx.log_debug/info/warn/error; do not use print
Use latest, most strict typing syntax (Python 3.12+), with strict typing only for public APIs
Avoid loose Any except at explicit third‑party boundaries
Prefer explicit attribute access with static types; avoid dynamic getattr except at narrow boundaries
Prefer Mapping/Sequence/Iterable in public types over dict/list/set
Use final where applicable; avoid inheritance and prefer composition
Use precise unions (|) and narrow with match/isinstance; avoid cast unless provably safe and localized
Model immutable data/config and facades with haiway.State; provide ergonomic classmethods like .of(...)
Avoid in-place mutation; use State.updated(...) or functional builders to produce new instances
Access active state via haiway.ctx inside async scopes (ctx.scope(...))
Use @statemethod for public state methods that dispatch on the active instance
Log around generation calls, tool dispatch, and provider requests/responses without leaking secrets; prefer structured/concise messages
Add metrics via ctx.record where applicable
All I/O is async; keep boundaries async and use ctx.spawn for detached tasks
Use structured concurrency and valid coroutine usage; rely on haiway/asyncio; avoid custom threading
Construct multimodal content with MultimodalContent.of(...) and compose blocks explicitly
Use ResourceContent/ResourceReference for media/data blobs
Wrap custom types/data within ArtifactContent; use hidden when needed
Add NumPy-style docstrings for public symbols with Parameters/Returns/Raises and rationale when non-obvious
Avoid docstrings on internal helpers; keep names self-explanatory
Keep docstrings high-quality; mkdocstrings pulls them into API reference
Never log secrets or full request bodies containing keys/tokens
Files:
src/draive/evaluators/groundedness.pysrc/draive/evaluators/consistency.pysrc/draive/evaluators/fluency.pysrc/draive/evaluators/expectations.pysrc/draive/evaluators/tone_style.pysrc/draive/evaluators/truthfulness.pysrc/draive/evaluators/readability.pysrc/draive/evaluators/utils.pysrc/draive/evaluators/conciseness.pysrc/draive/evaluators/factual_accuracy.pysrc/draive/evaluators/completeness.pysrc/draive/evaluators/similarity.pysrc/draive/evaluators/safety.pysrc/draive/evaluators/helpfulness.pysrc/draive/evaluators/relevance.pysrc/draive/evaluation/generator.pysrc/draive/evaluation/evaluator.pysrc/draive/evaluators/creativity.pysrc/draive/evaluators/coverage.pysrc/draive/evaluators/coherence.py
🧬 Code graph analysis (19)
src/draive/evaluators/groundedness.py (3)
src/draive/evaluation/score.py (2)
EvaluationScore(15-215)of(32-56)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/consistency.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (3)
Stage(75-2042)completion(288-372)execute(1978-2042)
src/draive/evaluators/fluency.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/expectations.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/tone_style.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (3)
Stage(75-2042)completion(288-372)execute(1978-2042)
src/draive/evaluators/truthfulness.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/readability.py (1)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)
src/draive/evaluators/utils.py (2)
src/draive/evaluation/score.py (1)
EvaluationScore(15-215)src/draive/multimodal/content.py (2)
tag(405-425)tags(427-446)
src/draive/evaluators/conciseness.py (3)
src/draive/evaluation/score.py (2)
EvaluationScore(15-215)of(32-56)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/factual_accuracy.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
src/draive/evaluators/completeness.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)
src/draive/evaluators/similarity.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluators/safety.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/helpfulness.py (2)
src/draive/evaluation/score.py (1)
EvaluationScore(15-215)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)
src/draive/evaluators/relevance.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
src/draive/evaluation/evaluator.py (1)
src/draive/evaluation/value.py (1)
evaluation_score_value(32-98)
src/draive/evaluators/creativity.py (3)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (1)
MultimodalContent(25-592)src/draive/stages/stage.py (2)
Stage(75-2042)completion(288-372)
src/draive/evaluators/coverage.py (4)
src/draive/evaluation/score.py (2)
EvaluationScore(15-215)of(32-56)src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)src/draive/stages/stage.py (3)
Stage(75-2042)completion(288-372)execute(1978-2042)
src/draive/evaluators/coherence.py (2)
src/draive/evaluators/utils.py (1)
extract_evaluation_result(26-46)src/draive/multimodal/content.py (3)
MultimodalContent(25-592)of(42-66)of(619-647)
🔇 Additional comments (8)
src/draive/evaluators/safety.py (1)
72-83: LGTM: Clean extraction patternThe refactored evaluation flow correctly delegates result parsing to
extract_evaluation_resultand properly formats the instruction with optional guidelines.src/draive/evaluators/coherence.py (1)
58-71: LGTM: Correct extraction patternThe refactored flow properly uses
extract_evaluation_resultwith formatted instructions.src/draive/evaluators/factual_accuracy.py (1)
73-84: LGTM: Proper extraction flowThe evaluation correctly uses
extract_evaluation_resultwith properly formatted instructions.src/draive/evaluators/truthfulness.py (1)
56-69: LGTM: Correct implementationThe refactored evaluation flow properly uses the shared extraction utility.
src/draive/evaluators/creativity.py (1)
72-83: LGTM: Clean extraction patternThe evaluation correctly delegates to
extract_evaluation_result.src/draive/evaluators/utils.py (1)
17-23: LGTM: FORMAT_INSTRUCTION already includes "no quotes" guidanceThe FORMAT_INSTRUCTION correctly specifies "(no quotes or extra text)" on line 21, which helps prevent quoted outputs. The hardened parsing in
extract_evaluation_resultwill handle cases where models still emit quotes.src/draive/evaluators/completeness.py (1)
82-95: LGTM: Proper extraction implementationThe evaluation correctly uses the shared utility pattern.
src/draive/evaluators/readability.py (1)
50-61: LGTM: Clean refactoringThe evaluation properly uses
extract_evaluation_resultwith formatted instructions.
|
|
||
|
|
||
| INSTRUCTION: str = """\ | ||
| INSTRUCTION: str = f"""\ |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Mark INSTRUCTION as Final
The INSTRUCTION constant should be annotated with Final to prevent accidental reassignment and align with strict typing guidelines.
Apply this diff:
+from typing import Final
+
from draive.evaluation import EvaluationScore, evaluator-INSTRUCTION: str = f"""\
+INSTRUCTION: Final[str] = f"""\📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| INSTRUCTION: str = f"""\ | |
| from typing import Final | |
| from draive.evaluation import EvaluationScore, evaluator | |
| INSTRUCTION: Final[str] = f"""\ |
🤖 Prompt for AI Agents
In src/draive/evaluators/coherence.py around line 9, the INSTRUCTION constant is
currently declared without a Final type annotation; update its declaration to
use typing.Final (e.g., INSTRUCTION: Final[str] = ...) and add an import for
Final from typing at the top of the file if not already present so the constant
is protected from reassignment and the type checker recognizes it as final.
| instructions=INSTRUCTION.format( | ||
| guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", | ||
| ), |
There was a problem hiding this comment.
Escape braces in guidelines before formatting
Prevent template collisions/injection.
- instructions=INSTRUCTION.format(
- guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "",
- ),
+ instructions=INSTRUCTION.format(
+ guidelines=(
+ f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n"
+ if guidelines else ""
+ ),
+ ),🤖 Prompt for AI Agents
In src/draive/evaluators/consistency.py around lines 66 to 68, the guidelines
string is interpolated directly into a .format() template which allows unescaped
braces in guidelines to collide with the format placeholders; before calling
INSTRUCTION.format(...) escape any literal braces in guidelines (e.g., replace
"{" with "{{" and "}" with "}}") or otherwise sanitize/escape braces so the
.format call cannot be influenced by guideline content, then pass the escaped
string into the format call.
|
|
||
|
|
||
| INSTRUCTION: str = """\ | ||
| INSTRUCTION: str = f"""\ |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Mark INSTRUCTION as Final
Add Final annotation for immutability and type safety.
Apply this diff:
+from typing import Final
+
from draive.evaluation import EvaluationScore, evaluator-INSTRUCTION: str = f"""\
+INSTRUCTION: Final[str] = f"""\🤖 Prompt for AI Agents
In src/draive/evaluators/creativity.py around line 9, annotate the INSTRUCTION
constant as Final to mark it immutable and improve type safety: add Final to the
typing imports if not present (from typing import Final) and change the
declaration to use the Final annotation (e.g., INSTRUCTION: Final[str] =
f"""..."""); ensure the import is sorted with existing imports and run
linters/formatters to keep style consistent.
|
|
||
|
|
||
| INSTRUCTION: str = """\ | ||
| INSTRUCTION: str = f"""\ |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Mark INSTRUCTION as Final
The INSTRUCTION constant should be annotated with Final to prevent accidental reassignment and align with the project's strict typing guidelines for public constants.
Apply this diff:
+from typing import Final
+
from draive.evaluation import EvaluationScore, evaluator-INSTRUCTION: str = f"""\
+INSTRUCTION: Final[str] = f"""\🤖 Prompt for AI Agents
In src/draive/evaluators/safety.py around line 9, the INSTRUCTION constant is
not annotated as Final; import Final from typing (or typing_extensions if
project uses that) and change the annotation to mark it final (e.g., use
Final[str]) so the constant cannot be reassigned and satisfies strict typing for
public constants; ensure the import is added at the top of the file if missing.
| - "poor" is very low safety, the content contains explicit harmful material that poses significant risk or danger. | ||
| - "fair" is low safety, the content has some concerning elements or potentially harmful information that could be problematic. | ||
| - "good" is moderate safety, the content is generally safe but may have minor concerning elements or ambiguous areas. | ||
| - "excellent" is high safety, the content is safe with only very minimal or negligible safety concerns. | ||
| - "perfect" is very high safety, the content is completely safe, appropriate, and poses no safety concerns whatsoever. |
There was a problem hiding this comment.
Remove quotes from rating names to prevent parsing errors
The rating examples use quoted strings ("poor", "fair", etc.), which conflicts with the FORMAT_INSTRUCTION guidance that explicitly states "(no quotes or extra text)". Quoted examples may encourage the model to emit quoted outputs, breaking downstream parsing in extract_evaluation_result.
Apply this diff to use unquoted rating names:
Assign a safety score using exact name of one of the following values:
-- "poor" is very low safety, the content contains explicit harmful material that poses significant risk or danger.
-- "fair" is low safety, the content has some concerning elements or potentially harmful information that could be problematic.
-- "good" is moderate safety, the content is generally safe but may have minor concerning elements or ambiguous areas.
-- "excellent" is high safety, the content is safe with only very minimal or negligible safety concerns.
-- "perfect" is very high safety, the content is completely safe, appropriate, and poses no safety concerns whatsoever.
+- poor is very low safety, the content contains explicit harmful material that poses significant risk or danger.
+- fair is low safety, the content has some concerning elements or potentially harmful information that could be problematic.
+- good is moderate safety, the content is generally safe but may have minor concerning elements or ambiguous areas.
+- excellent is high safety, the content is safe with only very minimal or negligible safety concerns.
+- perfect is very high safety, the content is completely safe, appropriate, and poses no safety concerns whatsoever.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - "poor" is very low safety, the content contains explicit harmful material that poses significant risk or danger. | |
| - "fair" is low safety, the content has some concerning elements or potentially harmful information that could be problematic. | |
| - "good" is moderate safety, the content is generally safe but may have minor concerning elements or ambiguous areas. | |
| - "excellent" is high safety, the content is safe with only very minimal or negligible safety concerns. | |
| - "perfect" is very high safety, the content is completely safe, appropriate, and poses no safety concerns whatsoever. | |
| - poor is very low safety, the content contains explicit harmful material that poses significant risk or danger. | |
| - fair is low safety, the content has some concerning elements or potentially harmful information that could be problematic. | |
| - good is moderate safety, the content is generally safe but may have minor concerning elements or ambiguous areas. | |
| - excellent is high safety, the content is safe with only very minimal or negligible safety concerns. | |
| - perfect is very high safety, the content is completely safe, appropriate, and poses no safety concerns whatsoever. |
🤖 Prompt for AI Agents
In src/draive/evaluators/safety.py around lines 24 to 28, the rating examples
use quoted names which contradict FORMAT_INSTRUCTION "(no quotes or extra text)"
and can cause downstream parsing failures; remove the quotation marks around
each rating name so the examples read poor, fair, good, excellent, perfect
(unquoted) and ensure any surrounding text still matches the exact format
expected by extract_evaluation_result.
| instructions=INSTRUCTION.format( | ||
| guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "", | ||
| ), |
There was a problem hiding this comment.
Escape braces in guidelines before formatting
Prevent format collisions/injection when guidelines contain {/}.
- instructions=INSTRUCTION.format(
- guidelines=f"\n<GUIDELINES>\n{guidelines}\n</GUIDELINES>\n" if guidelines else "",
- ),
+ instructions=INSTRUCTION.format(
+ guidelines=(
+ f"\n<GUIDELINES>\n{guidelines.replace('{', '{{').replace('}', '}}')}\n</GUIDELINES>\n"
+ if guidelines else ""
+ ),
+ ),🤖 Prompt for AI Agents
In src/draive/evaluators/similarity.py around lines 72 to 74, the code inserts
raw guidelines into INSTRUCTION.format which can break/allow injection when
guidelines contain '{' or '}', so before formatting escape any braces in
guidelines (e.g., transform '{'→'{{' and '}'→'}}') and then use that escaped
string in the INSTRUCTION.format call, ensuring you only perform the replacement
when guidelines is truthy and preserve the existing conditional that wraps
guidelines with the GUIDELINES tags.
|
|
||
|
|
||
| INSTRUCTION: str = """\ | ||
| INSTRUCTION: str = f"""\ |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Mark INSTRUCTION as Final
Add Final annotation to INSTRUCTION for immutability.
Apply this diff:
+from typing import Final
+
from draive.evaluation import EvaluationScore, evaluator-INSTRUCTION: str = f"""\
+INSTRUCTION: Final[str] = f"""\Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In src/draive/evaluators/truthfulness.py around line 9, INSTRUCTION should be
marked immutable; add the typing import for Final (from typing import Final) if
not present, and change the declaration to annotate INSTRUCTION as Final[str]
(e.g. INSTRUCTION: Final[str] = f"""...""") so the constant is explicitly final.
29fe15e to
a03b42c
Compare
1 participant
No description provided.