refactor(regex): one slugify, one skills tokenizer and one word tokenizer, the no-op ANSI copy gone, and the dead command-utils template retired (#18093, #18124)

[mrveiss]

Thinking Path

#18093 found six regex concepts, each forked across 2–8 files, with one fork a no-op copy. #18124 found a dead code-generation template. Both were finished from a local-only branch from 2026-10-10 and rebased onto current main. Five of the six concepts had already landed on main (ANSI, frontmatter, email/UUID, git ref, endpoint normaliser). This branch delivers the rest.

What Changed

• ANSI: the command-utils template emits the canonical ANSI stripper, not a no-op copy.
• Slug: one autobot_shared/slug.py sanitize_token with explicit parameters (allowed set, replacement, collapse, case, cap). The 12 migrated sites are each pinned byte-identical to their old output (utils/slug_equivalence_18093_test.py). Sites that can't migrate are on the guard's reasoned allowlist.
• Tokenizers: one skills tokenizer, and one word tokenizer shared by skills and fingerprinting (autobot_shared/text_tokens.py).
• tech-debt(code_analysis): a 190-line code-generation template whose output never landed, now stale against the file that did #18124: the dead command-utils code-generation template is removed. Its symbols have zero callers on HEAD; its output path src/utils/ doesn't exist, so it would have raised; and its purpose lives in utils/command_utils.py. A test (analyze_duplicates_template_18093_test.py) stops a generated copy coming back.
• canonical(knowledge): two functions named content_hash return different values — full digest vs 16-char truncation #15722: markdown_reference_system.py uses the one content_digest (byte-identical), and its queued guard entry is removed.
• Guard: main's regex_concept_fork_guard_18093_test.py is extended, not forked. It gains the word-tokenizer and char-sanitizer-class concepts with contrast fixtures and a non-vacuity test.

Verification

• Digest, regex-fork, symbol-fork, size, ceiling parity and guard-reach meta: 126 passed. reach_scope_claim_17844 run alone: 10.
• The three 18093 equivalence tests plus agent_loop: 288 passed. Skills, encoding, tool-output-filter, MCP aggregation, prepared facts and markdown-reference: 388 passed.
• Reversal check: main's later edits to fingerprint.py (content_digest), analytics_infrastructure.py (CollectionEmbeddingSpace), natural_language_search.py and doc_indexer.py are all kept; the branch adds only this intent.

Closes #18093
Closes #18124
Refs #15722

Model Used

Claude Opus 5.5 (master session: brief, PR). Finished by a Claude Sonnet subagent; the original commits came from a subagent on 2026-10-10.

🤖 Generated with Claude Code

[coderabbitai]

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 54 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

• Configuration used: Repository: mrveiss/AutoBot-AI/.coderabbit.yaml
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: 827bf470-527d-44c0-bbf0-6db23a0f032b

---

📥 Commits

Reviewing files that changed from the base of the PR and between f194061 and 495d61f.

---

⛔ Files ignored due to path filters (2)

• repo_tests/python_file_size_ratchet_baseline.py is excluded by !repo_tests/python_file_size_ratchet_baseline.py
• scripts/python_file_size_known_large.py is excluded by !scripts/python_file_size_known_large.py

---

📒 Files selected for processing (29)

• autobot-backend/agent_loop/fingerprint.py
• autobot-backend/agent_loop/text_utils.py
• autobot-backend/agents/browser/captcha_solver.py
• autobot-backend/agents/scope_enforcement.py
• autobot-backend/api/natural_language_search.py
• autobot-backend/code_analysis/scripts/analyze_duplicates.py
• autobot-backend/code_analysis/scripts/analyze_duplicates_template_18093_test.py
• autobot-backend/code_analysis/src/env_analyzer.py
• autobot-backend/code_intelligence/analytics_infrastructure.py
• autobot-backend/knowledge/backends/async_chromadb_adapter.py
• autobot-backend/knowledge/search_quality.py
• autobot-backend/llm_shared/tiered_routing/complexity_scorer.py
• autobot-backend/markdown_reference_system.py
• autobot-backend/prepared_facts.py
• autobot-backend/services/captcha_solver.py
• autobot-backend/services/knowledge/doc_indexer.py
• autobot-backend/services/mcp_aggregation.py
• autobot-backend/services/skill_management/skill_extractor.py
• autobot-backend/services/terminal_secrets_service.py
• autobot-backend/services/tool_output_filter.py
• autobot-backend/skills/builtin/skill_router.py
• autobot-backend/skills/text_tokens_equivalence_18093_test.py
• autobot-backend/utils/slug_equivalence_18093_test.py
• autobot-infrastructure/shared/tools/index_documentation.py
• autobot-slm-backend/api/venv_reconcile.py
• autobot_shared/slug.py
• autobot_shared/text_tokens.py
• repo_tests/one_content_digest_15722_test.py
• repo_tests/regex_concept_fork_guard_18093_test.py

---

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

✅ SSOT Configuration Compliance: Passing

🎉 No new hardcoded values of either class — ssot and other both block.

Known backlog in pipeline-scripts/hardcoded_values_baseline.txt is suppressed and tracked in #14371.