Skip to content

fix(agent-os): detect context-cued bare SSNs in redactor and adapter PII patterns - #3801

Closed
dylanyunlon wants to merge 3 commits into
microsoft:mainfrom
dylanyunlon:fix/context-cued-bare-ssn-evasion
Closed

dylanyunlon wants to merge 3 commits into
microsoft:mainfrom
dylanyunlon:fix/context-cued-bare-ssn-evasion

Conversation

@dylanyunlon

Copy link
Copy Markdown
Contributor

Closes #3592

Problem

CredentialRedactor.PII_PATTERNS and integrations.base.PII_PATTERNS both require a separator between digit groups so that bare nine-digit numbers (tracking numbers, ZIP+4, ABA routing numbers) do not hard-block at the MCP gateway. That requirement also lets a genuine SSN through when it is written without separators next to an explicit cue:

  • SSN: 745102386
  • ssn=745102386
  • social security number 745102386

None of these match at either detector today (verified against both patterns), so tool output carrying a cued bare SSN is neither redacted at the gateway nor blocked by the adapters.

Fix

Add a "US SSN (context-cued)" pattern to both detection sites. The pattern fires only when a case-insensitive cue keyword (ssn, social security, social security number/num/no/#, soc sec) appears immediately before the nine-digit run, keeping uncued bare digits as non-matches so the false-positive suppression from #3531 is preserved. Both detectors carry the identical regex to stay in lockstep (#3591 tracks their alignment).

Call chain (2 entry points, 2 source files, 2 test files)

Entry point Guard
CredentialRedactor.find_pii_matches() PII_PATTERNS["US SSN (context-cued)"].pattern
adapter PII scan (autogen/bedrock) integrations.base.PII_PATTERNS[1]

Files changed (5)

File Change
agent-governance-python/agent-os/src/agent_os/credential_redactor.py New US SSN (context-cued) CredentialPattern with cue-anchored regex
agent-governance-python/agent-os/src/agent_os/integrations/base.py Mirrored context-cued regex in PII_PATTERNS tuple (index 1)
agent-governance-python/agent-os/tests/test_credential_redactor.py 18 new tests (11 positive cued matches, 6 negative uncued stability, 1 matched_text assertion)
agent-governance-python/agent-os/tests/test_pii_patterns.py 9 new tests (5 positive cued matches, 4 negative uncued stability)
CHANGELOG.md ### Fixed entry under [Unreleased]

Tests

18 new tests over the two detection call chains:

  • test_context_cued_bare_ssn_is_detected (11 tests): every cue variant from the issue (SSN:, ssn=, social security number, Social Security, soc sec, socsec, etc.) with a bare nine-digit run.
  • test_context_cued_ssn_does_not_match_uncued_bare_digits (6 tests): uncued bare digits (tracking numbers, invoices, routing numbers, phone-like) are NOT matched — preserves fix(agent-os): detect separated SSN forms without matching bare nine digits #3531 FP suppression.
  • test_context_cued_ssn_match_captures_digits_only (1 test): the matched_text includes the digit run for redaction.
  • test_shared_context_cued_ssn_matches_cued_bare_digits (5 tests): adapter-side PII_PATTERNS[1] matches cued forms.
  • test_shared_context_cued_ssn_rejects_uncued_bare_digits (4 tests): adapter-side pattern rejects uncued forms.

All 125 existing + new tests pass.

Signed-off-by: dylanyunlon dogechat@163.com

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests size/M Medium PR (< 200 lines) labels Aug 21, 2026
@github-actions

Copy link
Copy Markdown

🟡 Contributor Check: MEDIUM

Check Result
Profile MEDIUM
Credential LOW
Overall MEDIUM

Automated check by AGT Contributor Check.

@github-actions github-actions Bot added the needs-review:MEDIUM Contributor check flagged MEDIUM risk label Aug 21, 2026
@github-actions

Copy link
Copy Markdown

PR Review Summary

Check Status Details
🔍 Code Review ⚠️ Missing No current-run comment
🛡️ Security Scan ⚠️ Missing No current-run comment
🔄 Breaking Changes ⚠️ Missing No current-run comment
📝 Docs Sync ⚠️ Missing No current-run comment
🧪 Test Coverage ⚠️ Missing No current-run comment

Verdict: ⚠️ AI review incomplete; ready for human review

AI review comments are untrusted advisory output. The summary reports workflow-generated completion status only, not model-authored pass/fail claims.

@MohammadHaroonAbuomar MohammadHaroonAbuomar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • CI 'Spell-check changed files' fails on this PR's own added lines: Spell-check changed files (unknown words in added lines: uncued, undelimited, socsec, ISSN, Confessn); Please add the terms to .cspell-repo-terms.txt (or reword) so the gate passes.

@dylanyunlon

Copy link
Copy Markdown
Contributor Author

Right, should have caught the spell-check gate before pushing. Confirmed all five words are flagged on the added lines: uncued, undelimited, socsec, ISSN, Confessn.

Fixed in 03cfdd2:

  • Added all five terms to .cspell-repo-terms.txt in alphabetical position. socsec is a cue keyword in the regex alternation, uncued/undelimited appear in test names and comments, and ISSN/Confessn are the negative test cases from the boundary fix in bbcf358.
  • No code change, dictionary only.

131 tests pass.

…PII patterns

CredentialRedactor.PII_PATTERNS and integrations.base.PII_PATTERNS both
require a separator between digit groups so that bare nine-digit numbers
(tracking numbers, ZIP+4, ABA routing numbers) do not hard-block at the
MCP gateway.  That requirement also lets a genuine SSN through when it
is written without separators next to an explicit cue:

    SSN: 745102386
    ssn=745102386
    social security number 745102386

Add a "US SSN (context-cued)" pattern to both detection sites.  The
pattern fires only when a case-insensitive cue keyword (ssn, social
security, social security number/num/no/#, soc sec) appears immediately
before the nine-digit run, keeping uncued bare digits as non-matches so
the false-positive suppression from microsoft#3531 is preserved.  Both detectors
carry the identical regex to stay in lockstep (microsoft#3591 tracks their
alignment).

Call chain (2 entry points, 2 source files, 2 test files):

  CredentialRedactor.find_pii_matches()
    -> PII_PATTERNS["US SSN (context-cued)"].pattern
  adapter PII scan (autogen/bedrock)
    -> integrations.base.PII_PATTERNS[1]

18 new tests across test_credential_redactor.py (11 positive, 6
negative, 1 matched_text assertion) and test_pii_patterns.py (5
positive, 4 negative).

Closes microsoft#3592

Signed-off-by: dylanyunlon <dogechat@163.com>
…n/ISSN

Addresses review feedback: the |ssn) branch lacked a leading word
boundary, so any word ending in "ssn" (Assn, ISSN, Confessn) cued the
detector and would hard-block at the gateway — re-breaking the microsoft#3531
FP-suppression goal.

Added (?<![A-Za-z]) before the alternation group in BOTH files
(credential_redactor.py and integrations/base.py).  Verified all 12
positive cue forms still match.

Tests: 3 new negative cases in test_credential_redactor.py (Assn, ISSN,
Confessn) + 3 matching cases added to test_pii_patterns.py non-match
tuple.  131 total tests pass.

Signed-off-by: dylanyunlon <dogechat@163.com>
Adds the five words that the Spell-check CI gate flags as unknown in
the PR's added lines (review feedback from MohammadHaroonAbuomar):

  - Confessn  (negative test: word ending in 'ssn')
  - ISSN      (negative test: acronym ending in 'ssn')
  - socsec    (cue keyword in the regex alternation)
  - uncued    (test name: _uncued_bare_digits)
  - undelimited (test comment describing bare digit runs)

No code change — dictionary only.

Signed-off-by: dylanyunlon <dogechat@163.com>
@dylanyunlon
dylanyunlon force-pushed the fix/context-cued-bare-ssn-evasion branch from 03cfdd2 to e6c38e1 Compare September 17, 2026 09:07

@MohammadHaroonAbuomar MohammadHaroonAbuomar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The PR no longer merges: GitHub reports CONFLICTING and a local git merge origin/main conflicts in agent-governance-python/agent-os/src/agent_os/integrations/base.py (PII_PATTERNS, lines 22-36 of the merged file). Main (#3532 fix) replaced the first SSN regex with re.compile(r"(?<![A-Za-z0-9])\d{3}[\s.-]\d{2}[\s.-]\d{4}(?![A-Za-z0-9])") plus a long comment; this branch keeps the old \b\d{3}[\s.-]?\d{2}[\s.-]?\d{4}\b line and inserts the cued pattern after it. Rebase onto current main (branch is 161 commits behind) and resolve by keeping main's separator-required line and comment, then the new context-cued re.compile(...) block directly after it, before the email regex. Verified: with exactly that resolution tests/test_credential_redactor.py and tests/test_pii_patterns.py pass on main (155 passed). The rebase also picks up main's typing-extensions==4.16.0 pin in agent-governance-python/requirements/ci-test.txt, which fixes the cannot import name 'sentinel' from 'typing_extensions' collection errors that failed test (agent-os, 3.11/3.12/3.13) on the previous head bbcf358.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation needs-review:MEDIUM Contributor check flagged MEDIUM risk size/M Medium PR (< 200 lines) tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Context-cued bare SSNs evade both the redactor and adapter SSN detectors

2 participants