Repository navigation
fix(evaluator): fail closed when external backend returns an error - #3008
Merged
Imran Siddique (imran-siddique) merged 1 commit intoJun 13, 2026
Merged
Imran Siddique (imran-siddique) merged 1 commit into
Imran Siddique (imran-siddique) merged 1 commit into
Conversation
…2992) Previously _evaluate_flat and _evaluate_rules silently skipped backends whose BackendDecision carried a non-None error field. If every registered backend errored, evaluation fell through to the configurable default action (which can be allow), converting a backend crash into a permit decision. Fix: invert the condition so a non-None error immediately returns a deny PolicyDecision with audit_entry["error"]=True and error_detail captured for post-incident investigation. Later backends in the list are not consulted. Adds three regression tests in test_policy_backends.py covering: - error backend denies instead of falling through to default - subsequent backends not reached after an error - healthy backend after YAML miss still returns its own decision correctly Security audit: docs/security/audits/2026-06-12-evaluator-backend-error-fail-closed.md Signed-off-by: Imran Siddique <imran.siddique@opaque.co> Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Imran Siddique (imran-siddique)
requested a review
from MohammadHaroonAbuomar
as a code owner
June 13, 2026 03:14
github-actions
Bot
added
documentation
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
PR Review Summary
Verdict: AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims. |
Imran Siddique (imran-siddique)
merged commit Jun 13, 2026
a1c6ff8
into
main
133 of 134 checks passed
Imran Siddique (imran-siddique)
deleted the
fix/evaluator-backend-error-fail-closed
branch
June 13, 2026 23:04
jlaportebot (jlaportebot)
pushed a commit
to jlaportebot/agent-governance-toolkit
that referenced
this pull request
Jun 17, 2026
…icrosoft#3008) Previously _evaluate_flat and _evaluate_rules silently skipped backends whose BackendDecision carried a non-None error field. If every registered backend errored, evaluation fell through to the configurable default action (which can be allow), converting a backend crash into a permit decision. Fix: invert the condition so a non-None error immediately returns a deny PolicyDecision with audit_entry["error"]=True and error_detail captured for post-incident investigation. Later backends in the list are not consulted. Adds three regression tests in test_policy_backends.py covering: - error backend denies instead of falling through to default - subsequent backends not reached after an error - healthy backend after YAML miss still returns its own decision correctly Security audit: docs/security/audits/2026-06-12-evaluator-backend-error-fail-closed.md Signed-off-by: Imran Siddique <imran.siddique@opaque.co> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Signed-off-by: jlaportebot <jlaportebot@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2992.
Summary
PolicyEvaluator._evaluate_flatand_evaluate_rulessilently skipped backends whoseBackendDecision.errorwas non-None, letting evaluation fall through to the configurable default action (which can beallow). A transient backend crash could produce a permit.PolicyDecisionwithaudit_entry["error"]=Trueanderror_detailcaptured for post-incident investigation. Subsequent backends in the list are not consulted.docs/security/audits/2026-06-12-evaluator-backend-error-fail-closed.md.Test plan
test_backend_error_fails_closed_not_fallthrough-- error backend denies, not falls through to defaulttest_backend_error_does_not_consult_subsequent_backends-- later backends not reached after first errorstest_healthy_backend_after_yaml_miss_still_allows-- non-error backend path still works correctlypython -m pytest agent-governance-python/agent-os/tests/ -q -k "evaluator"-- 116 passed (1 pre-existing crewai failure unrelated: missingagtcompiled extension)🤖 Generated with Claude Code