Skip to content

Fix SimpleOutput to drop annotations for items failing contains subschema#612

Open
staging-devin-ai-integration[bot] wants to merge 1 commit intomainfrom
evalon/blaze-the--d45b6269
Open

Fix SimpleOutput to drop annotations for items failing contains subschema#612
staging-devin-ai-integration[bot] wants to merge 1 commit intomainfrom
evalon/blaze-the--d45b6269

Conversation

@staging-devin-ai-integration
Copy link

Fix SimpleOutput to drop annotations for items failing contains subschema

Summary

The SimpleOutput collector was incorrectly retaining annotations (e.g. title) from array items that fail against a contains subschema. For example, given:

{ "contains": { "type": "number", "title": "Test" } }

and instance ["foo", 42, true], the title annotation was being kept for /0 and /2 even though those items don't match the contains subschema. Only /1 (42) should retain it.

Root cause: The mask system uses an early return to suppress errors within a contains context (since only one match is needed). However, this early return also skipped the annotation cleanup code, so annotations from failing items were never dropped.

Fix: Before the early return in the contains mask path (!entry.second), iterate annotations and erase those matching both the failing evaluate_path (via starts_with_initial) and the failing instance_location (via starts_with). This mirrors the existing cleanup logic but scopes it to the specific instance location that failed.

Three new tests are added covering:

  1. Mixed pass/fail items — only passing item retains annotations
  2. All items pass — all annotations retained (regression guard)
  3. All items fail — no annotations retained

Review & Testing Checklist for Human

  • Verify the instance_location.starts_with() filter doesn't over-erase annotations in nested contains scenarios (e.g. contains with object subschemas that produce annotations at sub-paths like /0/foo)
  • Consider whether minContains/maxContains edge cases need additional test coverage (same LoopContains handler is used, but no explicit tests added)
  • Confirm the annotation cleanup only fires for the contains mask (!entry.second) and doesn't affect anyOf/oneOf/not/if paths which use entry.second == true

Notes

…hema

The SimpleOutput collector was incorrectly retaining annotations from
items that fail against the contains subschema. This happened because
the mask-based early return for the contains context suppressed both
the error and the annotation cleanup.

The fix adds annotation cleanup within the contains context early
return path, filtering by both evaluate_path and instance_location.
This ensures annotations are only retained for items that actually
pass the contains subschema.

Co-Authored-By: bot_apk <apk@cognition.ai>
@staging-devin-ai-integration
Copy link
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants