Context enhancement is ineffective for agglutinative languages (e.g. Korean): lemma matching cannot see inside compound words

[juno-junho]

Is your feature request related to a problem? Please describe.

LemmaContextAwareEnhancer matches recognizer context words against per-token lemma strings. For Korean (spaCy ko_core_news_sm), lemmas are morpheme-joined strings like 급+여계+좌, so whether a context word survives inside that representation depends on where the statistical morpheme splitter happens to place its + boundaries.

The outcome is unpredictable even within the same word class. 계좌 (account) does match 계좌+이체 and 본인+계좌, but not 입금계+좌 (deposit account), 급+여계+좌 (salary account) or 회사+계+좌 (company account). 은행 (bank) matches 신한+은행 but not 국민은+행. It even fails for keywords standing completely alone: 주민등록번호 (resident registration number) lemmatizes to 주민등록번+호 and 예금주 (account holder) to 예금+주, so the keyword misses its own standalone occurrence.

I measured this end-to-end over 36 realistic Korean banking/PII sentences (benchmark in the comment below). The current enhancer fires on 16/36 (44%): 3/8 standalone, 7/10 with particles, 6/18 compounds. Raw-text substring matching fires on 36/36, with zero false boosts on the no-context controls.

For an English analogy: imagine compounds were written solid, like depositaccount, and your matcher only saw a statistical split like deposit+acc+ount. German compounds have the same shape (Konto inside Gehaltskonto). The existing :-split in NlpArtifacts.set_keywords doesn't help here, since ko uses + and splitting can't recover a keyword the segmentation cuts through. This also limits the effect of the Korean context terms recently added across kr_* recognizers (#1822, #1825, #2212, #2214).

Describe the solution you'd like

An opt-in SubstringContextAwareEnhancer that matches context words as raw-text substrings within a configurable character window (say ±100 chars) around each detection, keeping the same scoring semantics as the lemma enhancer (context_similarity_factor, min_score_with_context_similarity, MAX_SCORE cap). Cost is bounded by O(context words × window); repeated measurements show no overhead distinguishable from run-to-run noise (details in the benchmark comment). We've been running this in a production Korean PII-masking deployment.

Some design questions before I write the PR:

1. Configuration: the enhancer is currently replaceable only programmatically (AnalyzerEngine(context_aware_enhancer=...)). Would you accept an AnalyzerEngineProvider yaml key to select and parameterize the enhancer, with the default unchanged (lemma)?
2. Over-matching: the measurements below show the current default already substring-matches against lemma strings, so raw-text matching has the same English over-match profile (id inside video boosts in both), and the existing context_matching_mode="whole_word" suppresses it in both. Would a per-language matching mode (whole-word for space-delimited languages, raw-substring for agglutinative ones) be the preferred shape?
3. New class, or a raw-text fallback inside LemmaContextAwareEnhancer?

Describe alternatives you've considered

• Enumerating compound variants as context words: unbounded, doesn't scale.
• Splitting keywords on the ko + separator (like the existing : split): doesn't recover keywords the segmentation cuts through (급+여계+좌).
• Improving Korean lemmatization/tokenization: that's spaCy-model territory, and doesn't help other agglutinative languages.
• NER-based context detection: heavier and non-deterministic. The substring approach is a small, deterministic fix.

Additional context

We have a working, production-tested implementation and can submit a PR with tests once there's direction on the questions above. Benchmark data (hit rates, per-token lemma table, over-match comparison, latency) is in the first comment below. Related active work on the context path: #2208 (locale-aware case folding), #1969 (negative context).

[juno-junho]

Benchmark: how often does context enhancement actually fire on Korean text?

Setup: end-to-end through AnalyzerEngine (not a simulation of the matcher), spaCy ko_core_news_sm / en_core_web_sm, default enhancer parameters, pattern score 0.3 so a fired boost is unambiguous (0.3 → 0.65). Each case uses a single isolated context word, so every hit or miss is attributable to exactly one keyword against one surface form. No sampling involved, results are deterministic.

Korean hit rate (boost fired), current lemma matching vs raw-text substring:

category	n	lemma (current default)	raw-text substring
standalone keyword (계좌, 주민등록번호, ...)	8	3	8
keyword + particle (계좌로, 계좌번호는, ...)	10	7	10
keyword inside compound (입금계좌, 국민은행, ...)	18	6	18
total	36	16 (44%)	36 (100%)
no-context control sentences	3	0 false boosts	0 false boosts

(Raw-text substring hitting 36/36 is by construction, since every case's sentence verifiably contains the keyword in the raw text. What's being measured is how often the lemma path loses evidence that is present.)

Why the misses happen: the ko lemma is a morpheme-joined string, and boundary placement decides the outcome. Same word class, opposite results:

token	ko_core_news_sm lemma	keyword visible in lemma?
계좌이체	계좌+이체	O
본인계좌	본인+계좌	O
입금계좌	입금계+좌	X
급여계좌	급+여계+좌	X
신한은행	신한+은행	O
국민은행	국민은+행	X
주민등록번호 (standalone)	주민등록번+호	X
예금주 (standalone)	예금+주	X

This is also why splitting keywords on + (like the existing : split in NlpArtifacts.set_keywords) wouldn't fix it. The segmentation frequently cuts through the keyword itself, so no join/split recovers it.

On over-matching: raw-text substring is not a regression against the current default. The default mode already substring-matches context words against lemma strings (its own docstring gives 'lic' matches 'duplicate' as an example). Measured on English with isolated keywords and an SSN-shaped entity:

case	lemma-substring (default)	raw-text substring	whole_word mode
id inside video	false boost	false boost	no boost
id inside idea	false boost	false boost	no boost
lic inside duplicate	false boost	false boost	no boost
card inside discard	false boost	false boost	no boost
tin inside routine	false boost	false boost	no boost

The profile is identical in all five cases, and the existing context_matching_mode="whole_word" already suppresses all of them. That's why design question 2 in the issue body suggests a per-language matching mode rather than a new guard. One footnote: "my id is …" boosts in neither lemma mode, because en_core_web_sm tokenizes id as i + d. Raw-text matching catches it. Token-level matching is fragile even in English.

Cost: I originally reported a single run here (2.212 vs 2.277 ms); after repeating the measurement six times across two environments (PyPI release and main source), the honest summary is that both enhancers sit in the 2-4 ms range per short doc and the run-to-run variance is larger than any difference between them. Neither is consistently slower. The scan is bounded by the ±100-char window times the number of context words.

I can post the benchmark script if useful (single self-contained file). The PR is ready to shape according to the three questions in the issue body.

[juno-junho]

Benchmark harness: https://gist.github.com/juno-junho/52bced1572d9d935d17c8b7a1e406e69

Two files: context_bench.py and substring_context_aware_enhancer.py (a standalone copy of the raw-text enhancer, so nothing needs patching). Setup is in the docstring: install presidio-analyzer and the two spaCy sm models, put the files side by side, run. It prints each case with the token's actual lemma string, asserts the no-context controls, and writes the JSON behind the tables above.

The per-case context word is isolated on purpose, so every miss is attributable to one keyword against one surface form. If a different sentence mix would be more useful I can extend the corpus.

[juno-junho]

One more measurement, because the hit-rate numbers above understate the practical impact once a score threshold is in play.

Same sentence, same isolated context word 계좌, pattern score 0.3, boost +0.35:

문장: 급여계좌 110-234-567890 변경 신청합니다   (salary-account + bank account number)

threshold=0.0: lemma → 1 result (score=0.30) | raw substring → 1 result (score=0.65)
threshold=0.5: lemma → 0 results            | raw substring → 1 result (score=0.65)

At the default threshold (0) a lemma-mode miss only means a low-confidence score. But deployments that operate a threshold (which is what default_score_threshold exists for) lose the entity entirely: the bank account number above simply isn't returned, so downstream masking never sees it. That's how this surfaced in our production system in the first place.

Runnable with the two gist files from the previous comment plus a score_threshold argument on analyze().

[juno-junho]

An update since the benchmark comments: #2208 hit the same enhancer from the Turkish side, a locale-dependent case fold breaking context words at the comparison's first step the way segmentation breaks them at the second for Korean. Two language families now fail at different steps of the same comparison, which is what design question 2 (a per-language matching mode) was trying to get at.

Whenever you get to the three questions, the implementation is sitting on a branch and I would rather shape it to your answers than guess.