fix(scorers): detect the derivatives of a toxicity keyword

[KunyangZhang]

What

KeywordToxicityScorer missed the noun and adjective forms of its own keywords. Each of the 15 forms listed in #79 was reported as score=1.0, passed=True, assessed=True with details["toxic_categories"] == {}, so a response saying "the killer was identified" passed the scorer exactly the way "skills" does.

Fixes #79.

Part 1: derivatives of an existing keyword

_KEYWORD_DERIVATIVES maps each canonical keyword to a bounded, explicit set of derivative forms, and _keyword_pattern matches them as further alternatives of that keyword instead of adding entries to the keyword lists. That keeps every constraint from #79:

• the score denominator is still the 21 configured keywords, so one canonical hit still scores exactly 0.5238095238095238;
• details["toxic_categories"] still reports the canonical keyword - killer reports kill;
• a canonical keyword counts at most once per category per response, so "they killed ... and the killers escaped" is one match;
• aliases in different families still count separately;
• caller-supplied keywords, case-insensitive matching, the current boundaries and per-instance isolation are untouched. No stemming, no general suffix rule, and no return to substring matching.

Part 2: doubled consonants

_inflections() now repeats only the keyword's own final consonant instead of accepting any character in [bdglmnprt], so the synthetic killbed, killring and bombted no longer count, while stabbed, stabbing, bombed and bombing still do. The -ing e-dropping for hate is unchanged. Part 2 is tested separately from Part 1, so a passing derivative case cannot hide a suffix regression.

Regression plan

tests/test_keyword_toxicity_scorer.py gains 29 cases and modifies none of the existing ones:

• all 15 derivatives, each asserting stable canonical evidence, total_matches == 1, the exact score and the result flags - not merely that some match exists;
• the score denominator and the base-word score are unchanged;
• a base word plus several forms counts once; two families count twice;
• the existing benign controls (skills, skill set, stability, stable, offshoot, oxymoron, bombardier) stay clear;
• the synthetic doubled consonants are not hits; the valid doubled forms stay hits;
• an explicitly configured GuardedModel(output_scorers=[KeywordToxicityScorer()], block_on_scorer_fail=True) blocks "The killer was identified."

22 of the 29 new tests fail on main (e70df991) - verified by restoring rai_toolkit/scorers/programmatic.py from main, keeping the new test file, and rerunning it: 22 failed, 43 passed. tests/test_output_format_scorer.py passes in the same run, so the pre-fix failures come only from this change.

python -m pytest tests/test_keyword_toxicity_scorer.py -q     # 65 passed
python -m ruff check --select E9,F63,F7,F82 .                 # All checks passed!

The full suite is green on CI (all nine checks). Locally the only failure is tests/test_version.py::test_distribution_version_matches_public_import, which compares rai_toolkit.__version__ against the installed distribution metadata - this environment has an editable install built from an older wheel (0.2.0) while the source is 0.3.0, so it fails for reasons unrelated to this change.

One correction in this branch

The first commit rebuilt programmatic.py from a local checkout that predated #74, so it silently restored the old behaviour where an unsupported expected_format returned score=1.0, passed=True, assessed=True. CI caught it on six cases in tests/test_output_format_scorer.py; commit a89143d rebuilds the file from current main and re-applies only the #79 change, and the #74 contract is verified intact.

Scope

No configured keyword list changes, and the scoring formula is untouched. The formula question knisar raised on #73 - adding non-matching extra_keywords dilutes the score of a real hit - is filed separately as #91 with the reproduction and the contract options, so the formula can get an explicit decision. This change does not worsen it: the denominator is unchanged, and the alias forms do not enter it.

Notes: reuse==6.2.0 is not installable in this offline environment, so reuse --no-multiprocessing lint was not run locally; the test file keeps the SPDX header it already had. A ruff 0.16.7 is used locally rather than the 0.16.6 pinned in CI, on the CI's own E9,F63,F7,F82 selection.

This contribution was prepared with AI assistance, as the contributing guide asks contributors to disclose.

[knisar]

Thanks @KunyangZhang. The explicit derivative mapping keeps the configured keyword count stable, and the current diff preserves the OutputFormatScorer correction from #74. The latest hosted checks are green.

I found one case-sensitivity regression in the new doubled-consonant logic. With extra_keywords={"custom": ["nab"]}, scoring "nabbed" gives one match, score 0.5454545454545454, and passed=False. Change only the configured keyword to "NAB" and the same response gives zero matches, score 1.0, and passed=True. "GRAB" with "grabbing" has the same problem. Both uppercase forms match on main e70df99.

The membership check in _inflections() runs before the case-insensitive regex is compiled, so uppercase final consonants are excluded from the doubling alternatives. Please normalize the consonant check while preserving the caller's original keyword spelling in reported evidence. Add public-scorer regressions for lowercase, uppercase, and mixed-case custom keywords and response text, including doubled -ed and -ing forms. Keep the synthetic non-inflection controls and unchanged score denominator.

The formula decision in #91 remains separate and does not block this scoped fix. I verified the current and main scorer results with targeted offline checks; full-suite validation should be rerun after the correction.

[knisar]

[P2] Make the doubling eligibility check case-insensitive. _DOUBLING_CONSONANTS contains lowercase letters, so a configured NAB or GRAB never gets a doubled-consonant alternative. re.IGNORECASE cannot recover an alternative that was never built. This makes nabbed/grabbing pass with zero hits for uppercase custom keywords while lowercase configurations and current main detect them. Normalize this check and cover the public scorer with case variants, preserving the original keyword in evidence.

[KunyangZhang]

Fixed in e41ca34. _inflections() now tests keyword[-1].lower() in _DOUBLING_CONSONANTS, so "NAB", "GRAB" and mixed spellings get the same doubled alternative as the lowercase form. Only the test is normalized - the alternative is still built from keyword[-1], so re.IGNORECASE matches the response in any casing. Evidence keeps the caller's spelling: extra_keywords={"custom": ["NAB"]} reports {"custom": ["NAB"]}, and the denominator still counts one entry per configured keyword, so one hit with 22 configured keywords still scores 0.5454545454545454, as you measured.

While adding the case coverage I found the same mistake already on main in the e-dropping branch: it used keyword.endswith("e"), so a configured "SNARE" missed "snaring" for exactly the same reason ("snared" still matched, through the plain -ed). That check is normalized in the same commit. It is pre-existing rather than something this branch introduced, so say the word if you would rather have it split out.

Regressions added for lowercase, uppercase and mixed-case custom keywords and response text across both suffix paths - nabbed/NABBED/Nabbed/naBBed, grabbing/GrAbBing, snaring/SNARING - plus score identity across the three casings of one keyword, and KILLBED/KillRing/BOMBTED as case variants of the synthetic non-inflections that must stay clear.

Pre-fix check: with the a89143d file restored, 6 of the new cases fail (76 passed); with the fix the file is 82 passed.

[KunyangZhang]

Review-body items, all addressed in e41ca34:

• Case-sensitivity regression - fixed, see the inline thread.
• Regression coverage for lowercase / uppercase / mixed-case keywords and responses, doubled -ed and -ing, and the e-dropping form - added.
• Synthetic non-inflection controls kept and extended to uppercase spellings.
• Denominator unchanged - asserted explicitly: one hit with 22 configured keywords scores 0.5454545454545454 for all three casings.
• Full suite rerun after the correction: 564 passed, 15 skipped. Its one failure is tests/test_version.py::test_distribution_version_matches_public_import, which compares rai_toolkit.__version__ with the installed distribution metadata; the offline editable install here reports 0.2.0 while the source is 0.3.0, so it is an environment artifact rather than a result. python -m ruff check --select E9,F63,F7,F82 . passes.

#91 stays separate, as you said. I will follow up there with the acceptance table you asked for.

[knisar]

The doubled-consonant and e-dropping corrections work. I checked e41ca344 integrated with main 7ee275e in a clean Python 3.11 environment: 565 passed, 15 optional skips. Pinned lint, licensing, and patch whitespace checks passed. The latest hosted checks also pass.

One case-sensitive lookup remains in the new derivative path: _KEYWORD_DERIVATIVES.get(keyword, ()). Normalizing the suffix checks does not normalize this dictionary key.

Reproduction through the public scorer:

for keyword in ("kill", "KILL", "KiLl"):
    result = KeywordToxicityScorer(
        extra_keywords={"custom": [keyword]}, threshold=0.5
    ).score("killer")
    print(keyword, result.score, result.passed, result.details)

kill produces 2 matches, score 0.09090909090909094, and fails. KILL and KiLl produce only the built-in match, score 0.5454545454545454, and pass. The configured keyword's casing changes the verdict. The same issue affects all 15 derivative forms; 30 checks of uppercase/title-case configured keywords fail case invariance.

Please normalize the derivative lookup key while retaining the caller's original spelling in evidence. Add public-scorer regressions for lowercase, uppercase, and mixed-case configured keywords across derivative forms, asserting identical scores, match counts, and verdicts, plus original-spelling evidence. Keep the formula and #91 separate. The related SNARE e-dropping fix can stay in this PR.

[knisar]

Hi @KunyangZhang, following up on the September 29 review. The remaining blocker is the case-sensitive derivative lookup: normalize the lookup key while preserving the configured spelling in the evidence. Please add lowercase, uppercase, and mixed-case public-scorer regressions that retain identical scores, match counts, and verdicts. Keep the scoring-formula work in #91 separate.

Please share a brief status and your expected next update, or let me know if your availability has changed. Thanks for the work so far.