indicfuzz

Fuzzy name matching that actually works on Indian names.

The problem, stated concretely

Somewhere in a bank's KYC system, Kunhabdulla and Kunjabdulla are two customers. They are one man. His name has never been spelled the same way twice in his life, because the sound in the middle of it (ഞ, the palatal nasal) has no agreed English spelling. North Kerala writes it nh. Linguists write it nj. His passport says one, his PAN card says the other, and no software has ever noticed.

Here is what the standard tools make of that pair:

matcher	score	verdict
Levenshtein	2 edits	"fairly different"
Jaro-Winkler	0.94	same as it gives Rajesh vs Ramesh, who are two people
Soundex	K512 vs K512	correct, but it also matches Kunhabdulla to Kanabdool

They are all doing something reasonable. They are all wrong, because they were built for a writing system where spelling is roughly settled, and Indic romanization is not settled at all. There is no Pinyin for Malayalam. Every spelling is somebody's guess at a sound.

indicfuzz starts from the sound instead.

>>> import indicfuzz

>>> indicfuzz.ratio("Kunhabdulla", "Kunjabdulla")
1.0
>>> indicfuzz.ratio("Lakshmi", "Laxmi")
1.0
>>> indicfuzz.ratio("Choudhary", "Chowdhury")
0.9078

Same for the structural chaos that Indian records specialize in:

>>> indicfuzz.ratio("Kumar S.", "S. Kumar")            # nobody agrees on order
1.0
>>> indicfuzz.ratio("S. Ravi", "Sundar Ravi")          # Aadhaar vs PAN, forever
0.9597
>>> indicfuzz.ratio("Mohammed bin Rashid", "Mohd. Rasheed")
0.9978

And it does not care which script you hand it:

>>> indicfuzz.ratio("കുഞ്ഞബ്ദുള്ള", "Kunhabdulla")
1.0
>>> indicfuzz.ratio("राजेश कुमार", "Rajesh Kumar")
0.9932

Crucially, it still says no when the answer is no:

>>> indicfuzz.ratio("Rajesh", "Suresh")
0.413
>>> indicfuzz.ratio("Lakshmi", "Abubakar")
0.0

Scores are floats from 0.0 to 1.0. rapidfuzz returns 0 to 100, so if you are swapping one for the other, mind the scale.

Install

pip install indicfuzz

Pure Python, zero runtime dependencies, 3.9 and up. It will run anywhere you can get a Python interpreter, including the locked-down bank VM where you cannot install a compiler.

Scripts it reads

Malayalam, Devanagari, Tamil, Telugu, Kannada, Bengali, Gujarati, Gurmukhi, Oriya, Latin.

Native and romanized forms compare directly. There is no transliteration step to install, configure, or keep a model file for. Mixed input like "Rajesh കുമാർ" gets routed word by word, which happens more often than you would like in real data.

Searching a collection

Scoring a query against every row in a watchlist is not viable. Block first: a cheap phonetic key narrows the field, then only the survivors get scored properly.

from indicfuzz import NameIndex

index = NameIndex()
index.add("Kunhabdulla Musaliar", payload={"id": 1})
index.add("Rajesh Kumar", payload={"id": 2})

for hit in index.search("Kunjabdulla Musliyar", score_cutoff=0.8):
    print(hit.name, hit.score, hit.payload)

Every name goes into the index under one key per token plus a whole-name key. That token-level indexing is what lets S. Ravi find Sundar Ravi: their full-name keys look nothing alike, but they collide on Ravi.

Got a short list already? extract and extract_one just score everything and skip the index.

Speed, measured rather than promised

300,000 synthetic multi-token names, single core:

index build	~19,000 names/sec
candidates examined per query	~4.5% of corpus
indexed search	~3.6 s/query
unblocked linear scan	~79 s/query
raw scoring throughput	~3,800 comparisons/sec

Blocking buys about 22x. That is not enough yet, and pretending otherwise would waste your time.

Two things pull in opposite directions here. The test corpus is deliberately nasty (built from a small pool of stems, so buckets are much denser than real name distributions), which makes 4.5% a pessimistic number. But the actual ceiling is the scorer at 3,800 comparisons per second, and no blocking strategy rescues you from that.

Batch deduplication overnight: fine. Real-time payment screening: not yet. A compiled core is the honest prerequisite, not a nice-to-have. Until then, NameIndex(max_bucket_fraction=...) lets you trade recall for latency.

Does it actually work

On the 196-pair development set in benchmarks/, all matchers seeing the exact same pairs:

matcher	ROC AUC	best accuracy	precision @ 90% recall
indicfuzz	0.987	0.944	0.957
rapidfuzz token_sort_ratio	0.699	0.694	0.505
rapidfuzz ratio	0.669	0.673	0.505
Jaro-Winkler	0.602	0.612	0.505

Now the caveat, because these numbers are softer than they look. That set is hand-written, not sampled from real records, and the model was tuned against it. Numbers produced that way are optimistic about data they have not seen, by an amount nobody can estimate from the inside.

What the set does prove: the ranking is better, since every matcher scored identical pairs. What it does not prove: that you will see 0.944 on your database. Do not put these figures in a pitch deck.

The real benchmark comes from Aksharantar. That work is written up in benchmarks/README.md and is the next milestone.

python benchmarks/evaluate.py --by-category --errors

How it works

Four stages, each fixing something the previous one cannot see.

1. Tokenize into sounds

ksh and x both become (KA, SSA). All of nh, nj, ny, gn become NYA. Native script gets read with proper abugida rules: inherent vowels, virama, matras, chillu letters, nukta. Both paths end at the same phoneme stream, which is the whole trick behind cross-script matching.

2. Score substitutions by confusability, not by anatomy

This is the part people get wrong, including me on the first pass.

The intuitive move is to measure how far apart two sounds are in the vocal tract. Resist it. What you actually want to know is how likely two sounds are to be spelled the same way, and those two questions have different answers.

Dental /t/ and retroflex /ʈ/ are contrastive phonemes. Malayalam speakers hear them as clearly different. But English orthography writes both as t, every single time, so for name matching they are nearly free to swap. Meanwhile /k/ and /p/ sit no further apart on a feature chart and nobody in recorded history has confused them in spelling.

So the tables in phonology.py are confusion matrices, not anatomy. And the dimensions multiply rather than add:

distance = 1 - (1-place) x (1-manner) x (1-voicing) x (1-aspiration)

Why multiply? Try the additive version on /r/ and /s/. Both alveolar, so the place term contributes nothing, and you get about 0.28. Which says Rajesh and Sajesh are near-identical names. Multiplying lets one maximal difference dominate, the way it should, while leaving the small stuff untouched: k/g still costs 0.12, k/kh still costs 0.08.

3. Align the sequences

Weighted edit distance, with two adjustments that matter.

Vowels count for 45% of a consonant. Indic vowel romanization is close to lawless (Deepak, Dipak, Deepack) and weighting vowels equally drowns the signal that consonants carry.

Cost also decays toward the end of a word. Dravidian names take productive nominative suffixes that databases record however they feel: Ganesh and Ganesan, Krishna and Krishnan, Sudhakar and Sudhakaran. One name each, and the difference is always at the end.

4. Fix the structure

Parse into tokens, drop honorifics and relational particles, expand clerical abbreviations, then match the two token sets by optimal assignment instead of by position.

Word order carries zero weight, which is correct for a naming system where the same person's tokens land in different orders in different databases. This is also where initials get handled: S. is consistent with Sundar because they open on the same phoneme, and consistency scores 0.90 rather than 1.0, because S. is equally consistent with Suresh and Sreekumar.

Things it refuses to do

No name-equivalence table

An earlier version of this library shipped a lookup table claiming Varghese = Varkey = Vareed = Geevarghese, Thomas = Ouseph, John = Ninan, Krishnan = Kishan.

Every one of those is wrong. They are different names. Some of them belong to different people in the same family.

The reason this matters more than it looks: the two failure modes are not symmetric. Miss a spelling variant and you lose recall, which a lower threshold gets back. Assert a false equivalence and you have told the system that two people are one person, at score 1.0, sitting above every threshold anyone can set. In sanctions screening or civil registry deduplication, that is the error that ends careers.

So there is no table. Spelling variants get handled phonetically, because phonetically they are nearly identical. Names that merely coexist in a community do not get merged, because phonetically they are not. If your deployment genuinely needs domain equivalences, put them in your config where someone owns the consequences.

Same logic caps the abbreviation list at forms that are never themselves a name: Mohd, Md, Muhd expand to Mohammed. Raj does not expand to Rajan, because Raj is a name.

No honorific stripping that eats real names

The honorific list once included Kumari, Beevi, Musaliar, Thangal, Sahib, Haji. Every one of those is a title in some records and part of somebody's name in others.

The damage was easy to miss and thoroughly stupid: Kumar versus Kumari scored 0.000, because Kumari got stripped to nothing and the comparison ran against an empty string. Fathima Beevi lost her name to a title list.

The test for inclusion now is whether anyone is named this. If yes, it stays, and the ordinary half-weight penalty for a missing token handles its inconsistent presence.

Known limitations

Measured, not hypothetical. Each one is pinned by an assertion in benchmarks/check_regression.py, so it cannot quietly get worse.

Gender-suffix pairs score too high. Kumar versus Kumari lands at 0.923. Shyam versus Shyamala is similar. They differ by a final vowel, and the position weighting that correctly forgives Ganesh/Ganesan has no way to tell a gender suffix from a nominative one. Phonetics cannot fix this. The information you need is a gender field, which lives in the record and not in the string. Filter downstream if you have it.

Frequent tokens are not discounted. Nair Rajesh versus Nair Ramesh scores 0.87 because the shared surname dilutes the token that actually differs. Proper record linkage weights tokens by inverse frequency. That needs corpus statistics this library does not ship.

The positive and negative bands still overlap by roughly 0.15. No single threshold cleanly separates them, so this is not a drop-in answer for zero-tolerance screening. Closing that gap is the main open problem.

Scores are not probabilities. 0.85 does not mean 85% likely the same person. Pick thresholds against your own labelled data.

Tamil is lossy by design of the script itself. It marks neither voicing nor aspiration, so க covers /k/, /g/ and /h/. Decoding takes the unvoiced value and leans on a low voicing penalty to absorb the rest.

No Perso-Arabic support. Names written in Urdu or Arabic script are out of scope, which means the Gulf triangulation case (Indian passport, Emirates ID, English sanctions list) is not addressed yet.

The weights are hand-set, not learned. See below.

Where this goes next

The highest-value thing left is deleting my judgment from phonology.py.

Right now the confusion matrices encode a reasonable person's intuitions about Indic orthography. That is better than the alternatives, and it is still somebody's opinion. Aksharantar ships 26 million transliteration pairs. Count how often two romanizations line up against the same native grapheme, take -log P(same phoneme), and the entire table falls out of evidence instead of argument.

The model stays fully deterministic and auditable. It is still a lookup table. It just stops being my lookup table. Nothing outside phonology.py cares where the numbers came from, which was the point of putting them all in one place.

After that: IDF token weighting, score calibration, a Rust core once the algorithm stops moving, and Perso-Arabic script.

Development

pip install -e ".[dev,benchmark]"
pytest                                   # 272 tests, doctests included
python benchmarks/evaluate.py            # quality report
python benchmarks/check_regression.py    # CI gate

The benchmark runs in CI as a gate, not as decoration. Matching quality is the product, so a refactor that leaves every unit test green while quietly moving AUC from 0.98 to 0.85 should fail the build. Nothing else in the suite would catch it, since the individual assertions are all about specific pairs rather than the shape of the score distribution.

License

MIT.