fix: propagate api error message

[asamal4]

Propagate API error to the output.

Summary by CodeRabbit

• Bug Fixes

  • Clearer error messages when evaluation API steps fail, with metrics consistently marked as ERROR and accompanied by the specific reason.
  • Eliminates misleading success indications in rare API failure scenarios.

• Refactor

  • Standardized error propagation in the evaluation pipeline to use message-based reporting for improved reliability and diagnostics.

[coderabbitai]

Walkthrough

Return type of APIDataAmender.amend_conversation_data changed from bool to Optional[str], shifting error signaling to a returned message. Processor updated to consume the optional error message: if present, it logs and marks metrics as ERROR; if None, it proceeds with evaluation. No other public API changes.

Changes

Cohort / File(s)	Summary
API amendment return-type change src/lightspeed_evaluation/pipeline/evaluation/amender.py	Change method signature to return Optional[str] instead of bool. Return None for no error; return a string message on error. Adjust early returns and exception paths to provide error text.
Processor error handling adaptation src/lightspeed_evaluation/pipeline/evaluation/processor.py	Replace boolean api_error_occurred with api_error_message from amend_conversation_data. If a message is returned, log and mark all metrics as ERROR with that message; otherwise continue with per-turn and per-conversation evaluation.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Runner as EvaluationRunner
  participant Processor as Processor
  participant Amender as APIDataAmender
  participant API as api_client

  Runner->>Processor: process(conversation)
  Processor->>Amender: amend_conversation_data(conv_data)
  Amender->>API: call external API
  alt API success
    API-->>Amender: response OK
    Amender-->>Processor: None (no error)
    Processor->>Processor: proceed with metrics evaluation
  else API error/exception
    API-->>Amender: error
    Amender-->>Processor: "error message"
    Processor->>Processor: mark all metrics = ERROR (message)
  end

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Poem

A whisk of code, a twitching ear,
Errors now speak, their messages clear.
If silence returns, we hop right through—
Metrics parade in orderly queue.
I nibble the logs, tail in delight,
Optional whispers guide us right. 🥕🐇

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title clearly states that the pull request implements propagating API error messages rather than a boolean flag, which directly reflects the changes made to amend_conversation_data and error handling in processor.py. It is concise and specific without unnecessary details or noise. This makes the intent immediately understandable to reviewers.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

📜 Recent review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 94c8236 and d527cf4.

📒 Files selected for processing (2)

• src/lightspeed_evaluation/pipeline/evaluation/amender.py (2 hunks)
• src/lightspeed_evaluation/pipeline/evaluation/processor.py (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (2)

src/lightspeed_evaluation/pipeline/evaluation/processor.py (2)

src/lightspeed_evaluation/pipeline/evaluation/amender.py (1)

• amend_conversation_data (20-71)

src/lightspeed_evaluation/pipeline/evaluation/errors.py (1)

• mark_all_metrics_as_error (17-77)

src/lightspeed_evaluation/pipeline/evaluation/amender.py (1)

src/lightspeed_evaluation/core/models/data.py (1)

• EvaluationData (135-182)

🔇 Additional comments (5)

src/lightspeed_evaluation/pipeline/evaluation/amender.py (3)

62-69: LGTM! Error messages now propagate with context.

The error messages include turn_id and exception details, which will help with debugging. The logging at error level is appropriate.

---

71-71: LGTM! Success case correctly returns None.

Returning None on success is consistent with the new return type semantics documented in the function signature.

---

20-27: No further updates needed for callers of amend_conversation_data.
The only usage in processor.py already checks if api_error_message: and handles the returned string correctly.

src/lightspeed_evaluation/pipeline/evaluation/processor.py (2)

94-99: LGTM! Correctly consumes the new return type.

The variable api_error_message is properly initialized to None and captures the return value from amend_conversation_data. The assignment correctly reflects the new Optional[str] return type.

---

102-110: LGTM! Error propagation logic is sound.

The truthy check if api_error_message: correctly handles both None (success, falsy) and non-empty error strings (failure, truthy). The error message is appropriately passed to mark_all_metrics_as_error for detailed error tracking.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[asamal4]

@VladimirKadlec @tisnik PTAL
This should resolve misleading API error message.

[VladimirKadlec]

LGTM

[tisnik]

LGTM