Skip to content

Fix SimpleOutput annotation pruning for contains failures#607

Open
staging-devin-ai-integration[bot] wants to merge 1 commit intomainfrom
evalon/blaze-the--db99da59
Open

Fix SimpleOutput annotation pruning for contains failures#607
staging-devin-ai-integration[bot] wants to merge 1 commit intomainfrom
evalon/blaze-the--db99da59

Conversation

@staging-devin-ai-integration
Copy link

Fix SimpleOutput pruning of annotations in failing contains items

Summary

This PR fixes a SimpleOutput annotation collection bug where annotations produced while evaluating a contains subschema were retained for array items that failed the contains subschema.

What changed

  • SimpleOutput now prunes annotations using both:
    • evaluate path, and
    • instance location
      instead of evaluate path alone.
  • For failures masked under contains (which are intentionally not emitted as output errors), SimpleOutput now still performs annotation cleanup on Post failure events before returning.
  • Added a regression test covering:
    • contains with nested title
    • mixed array instance ["foo", 42, true]
    • verifies title annotation is kept only for /1 (the matching item) and dropped for /0 and /2.

Review & Testing Checklist for Human

  • Confirm the new pruning condition (evaluate_path + instance_location) does not accidentally preserve/drop annotations in non-contains failure cases (especially nested schemas and sibling annotation keywords).
  • Verify the masked-contains control flow change is correct: failures remain suppressed in output, but annotations from failing items are removed.
  • Reproduce the original CLI example (contains + title + mixed array) and confirm only the matching item retains the title annotation in verbose output.

Notes

Co-Authored-By: bot_apk <apk@cognition.ai>
@staging-devin-ai-integration
Copy link
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants