Skip to content

Conversation

@Mukhsin0508
Copy link

@Mukhsin0508 Mukhsin0508 commented Nov 14, 2025

Problem

TaskEvaluation Pydantic validation fails when LLM streaming responses omit the required quality field, causing memory save failures. This matches the issue described in #3915.

Solution

  • Made quality field optional with default value of 5.0
  • Improved evaluation prompt to emphasize quality requirement
  • Added comprehensive tests

Testing

  • Added tests for missing quality field (defaults to 5.0)
  • Added tests for provided quality field (backward compatible)
  • Verified no linting errors

Impact

  • Fixes validation errors preventing memory saves
  • Backward compatible (existing code continues to work)
  • Low risk (LongTermMemoryItem already accepts None for quality)

Fixes #3915


Note

Set a 5.0 default for TaskEvaluation.quality, tighten the evaluation prompt requirements, and add tests covering missing/provided quality and JSON parsing.

  • Utilities
    • TaskEvaluation: set quality default to 5.0 and update its description.
  • Evaluator Prompt
    • TaskEvaluator.evaluate: require ALL fields; emphasize numeric, required quality score in instructions.
  • Tests
    • Add tests verifying default quality when omitted, handling when provided, and model_validate_json with partial JSON; update imports accordingly.

Written by Cursor Bugbot for commit 5d1f444. This will update automatically on new commits. Configure here.

- Add default value of 5.0 to quality field in TaskEvaluation model
- Improve evaluation prompt to emphasize quality field requirement
- Add tests to verify default behavior and backward compatibility

Fixes validation errors when LLM output omits quality field during
streaming, preventing memory save failures in long-term storage.

Fixes crewAIInc#3915
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] ERROR:root:Failed to parse structured output from stream: 1 validation error for TaskEvaluation quality

1 participant