Record matching can improve PII redaction when a service already knows which customer is speaking. The service can match selected fields from that customer's record, combine those spans with a general PII detector, and redact the text before it reaches a model, analytics system, or log.
This repository provides a deterministic synthetic check of that design. It compares the same record matcher with two general detectors:
- transparent regular expressions adapted from DataFog at a pinned commit;
- Microsoft Presidio 2.2.364 with the pinned spaCy
en_core_web_sm3.8.0 model.
Each general detector runs alone and in combination with record matching. A separate control deliberately supplies the wrong customer record.
The generator uses reserved example domains, fictional 555-01xx phone numbers, test payment cards, and SSN area numbers 900 through 999, which the Social Security Administration excludes from assignment (SSA).
This is a sanity check, not a performance benchmark. The data are synthetic. The result establishes only that record matching and general detection have complementary error surfaces. Record matching is strongest where the organization already knows the answer. General detection is necessary where it does not.
The deterministic corpus contains 64 customer profiles, 512 chats, and 1,600 labeled PII mentions. Pooled across the canonical and stress suites, record matching raises Presidio's full-mention recall from 61.8% to 87.3%. Recall on record-derived mentions rises from 49.7% to 95.3%; recall on novel mentions remains 77.1%.
A mention counts as fully masked when every letter and digit in its gold span is masked. Spaces and punctuation between detected name or address components do not decide the result. Strict entity scores still require an exact span and label.
| Method | All mentions | Record-derived | Novel |
|---|---|---|---|
| Regex | 48.0% | 28.6% | 72.7% |
| Regex + record | 84.0% | 92.9% | 72.7% |
| Presidio | 61.8% | 49.7% | 77.1% |
| Presidio + record | 87.3% | 95.3% | 77.1% |
Supplying the wrong customer record removes the record-matching gain. The research note documents the corpus, methods, metrics, results, and limits.
The idea is not new. A 2008 clinical de-identification system matched patient names from the structured record linked to each note back into the note text, alongside dictionaries, regular expressions, and context rules. Its authors found that a generic spell checker added little sensitivity and many false positives (Neamatullah et al., 2008). A later system created patient-specific runtime dictionaries and combined them with learned and rule-based components (Dehghan et al., 2015). A 2016 system treated structured patient fields as a spelling-variant detection problem and used minimum edit distance with regular expressions on clinical text containing fragments, abbreviations, and misspellings (Yuwono et al., 2016).
This repository applies the same basic design to customer-support text and makes the comparison reproducible. It does not claim a new de-identification method or measure how often private systems use one.
The record matcher uses field-specific rules rather than applying fuzzy search to every value. It normalizes common phone, date, address, order, and account formats. For full names containing at least seven letters, it accepts a Levenshtein distance of one while requiring aligned name components to retain their initials; missing or substituted delimiters are also accepted. It permits one edit in a street name when the house number and suffix agree exactly. When the matching city, state, and ZIP follow that street line, the matcher masks the complete address; otherwise it masks only the supplied street line. The implementation uses RapidFuzz 3.14.5, pinned in the lock file.
Fuzzy matching is not applied to email addresses, dates, phone digits, order numbers, or account numbers. Those fields require an exact value after field-specific normalization. The matcher also refuses first names, ZIP codes, last-four values, incomplete dates, and other short fragments. These restrictions are deliberate: broad approximate matching can turn ordinary text into false positives.
The corpus is generated by deterministic formulas, not sampled from real customers or from a population model. Sixteen first names, sixteen last names, sixteen street names, four cities, and four street suffixes are combined by fixed index rules into 64 unique profiles. Email addresses use reserved example domains. Phone numbers use the fictional 555-01xx range. Payment cards are standard test numbers, and SSNs use unassignable area numbers.
Each profile produces eight chats. Four canonical chats use common renderings of known record fields and novel PII. Four stress chats include a reversed name with one character deleted or substituted, a hash-prefixed order number, a slash-separated phone number, an uppercase email address, a street line with one character error, an incomplete date of birth, third-party PII, and numeric distractors. Half of the mistyped street lines omit the city, state, and ZIP code; half retain the full address.
The generator labels each mention while assembling the text, so character offsets do not depend on a second detector. Every gold span is marked as known if its value derives from the resolved record or novel if it first appears in the chat. The stress suite was written after inspecting the matcher, so it is an adversarial software test rather than a held-out evaluation set.
Install Python 3.11, 3.12, or 3.13 and uv 0.12.5, then run:
uv sync --locked --all-extras --all-groups
make analysisThe command regenerates the profiles, chats, labels, predictions, redacted text, aggregate CSV files, and TeX tables in build/analysis/.
Run the complete local checks with:
make cimake ci requires a TeX distribution with latexmk. The standard Docker check covers formatting, linting, tests, and analysis without requiring TeX on the host:
make ci-dockerOther useful targets are make paper, make package, and make clean.
The full comparison, including Presidio, is:
uv run --extra benchmark self-redaction --presidio --output-dir build/analysisOmit --presidio for the lightweight regex and record-matching check.
The output directory contains:
synthetic_data.json, with every generated profile, chat, and gold span;chat_results.csv, with predictions and redacted text for every method and chat;summary.csv, with masking and strict entity metrics;stratified_recall.csv, split by known or novel source and entity type;record_resolution_summary.csv, with correct-record and wrong-record results;- generated TeX inputs used by the paper.
Record matching cannot find a new phone number, a third party's name, or any fact absent from the resolved record. General detection remains necessary. Record matching can still cover account IDs, order IDs, addresses, and other values that a general detector must infer from text.
Correct customer resolution is a security precondition. The matcher should retrieve only fields authorized for the current interaction, run inside the trusted redaction boundary, retain match provenance, and discard its temporary lookup values afterward. The customer record should not be copied into a model prompt.
Masking direct identifiers is not anonymization. This project does not measure re-identification risk, fairness, production latency, legal compliance, multilingual text, or speech-recognition output.
The project is released under the MIT License. LICENSES/DataFog.txt records the pinned DataFog source and its MIT notice. Presidio is MIT licensed, spaCy is MIT licensed, and the English spaCy model is MIT licensed.