refactor(dedup): eliminate code duplication via behavior-preserving extraction (#9794) by mrveiss · Pull Request #9865 · mrveiss/AutoBot-AI

mrveiss · 2026-06-10T09:07:57Z

Thinking Path

Ran the authoritative detector (jscpd/cpd 5.x, -k 70 -l 8 --skip-comments, tests/generated/migrations/mocks/i18n excluded) and worked the report largest-cluster-first. Every code change is behavior-preserving and verified against the existing test suite; skips were taken whenever an extraction required unverifiable guarantees, touched broken/unwired code, or would couple unrelated domains. No feature work mixed in. One commit per extraction.

What Changed

Duplication: 1.16% → 1.04% · −1,800 duplicated lines · 25 files (+303 / −2,232).

#	Cluster	Action	Result
1	`extensions/` vs `middleware/`	Completed the half-finished #7426 rename: converted the 6 leftover full-copy modules (`base`/`hooks`/`manager`/`hook_invoker`/`extension_manifest`/`builtin/__init__`) to re-export shims; moved `transcriber_extension` to canonical `middleware/builtin/`	−~1,453 lines
3	`gdrive` / `onedrive` (+`nextcloud`/`gitlab`)	Extracted byte-identical text-extraction helpers into `knowledge/connectors/content_extraction.py`	−~60 lines
5	`autobot-backend/static/{js,assets}`	Untracked stale vite build artifacts; fixed root-anchored `.gitignore` patterns that never matched the subdir	−541 tracked lines
6	CI	`duplication-guard.yml` — fails when duplication exceeds the 1.05% baseline (ratchet down only)	new gate
7	Docs	`DEDUP_REPORT.md` — per-cluster detail, before/after, skips	new

Public APIs preserved exactly — extensions.base.X, extensions.builtin.*, connector call-sites all still resolve via shims/aliases.

Deliberately skipped / deferred (documented)

Cluster 2 — code_intelligence/security_analyzer.py vs security/: SKIPPED. The security/ package (Bug Prediction Analysis: 119 Medium-Risk Files Identified (2026-01-12) #712 modularization) is broken (constants.py imports a removed DEBUG_MODE_VARS), unwired, and untested; the monolith is the live SSOT. Filed code_intelligence/security/ (#712 modularization) is broken & unwired — duplicate of live security_analyzer.py #9856.
Cluster 4 — Vue (9,666 dup lines, the largest source): DEFERRED. Almost entirely scoped-CSS / template clones with no unit tests and no visual-regression net, so visual identity can't be evidenced headless; partial CSS extraction is cascade-order sensitive. Filed umbrella Vue frontend duplication campaign — 9,666 dup lines across 274 component pairs (CSS/HTML/TS) #9859 with the safe migration methodology (BaseModal + design-system already exist).
Connector _classify_change/OAuth _request (depend on diverging _load_ts) and cross-layer sanitizer/precommit overlaps — left as-is to avoid wrong abstractions.

Verification

extensions/ + middleware/ + transcriber + gdrive + onedrive tests = 155 passed (post-rebase).
Connector batch suite 33 passed; import/identity parity smoke green.
Rebased onto updated Dev_new_gui (b25418f95); one conflict in feature_routers.py resolved by accepting upstream's fix(api-wiring): resolve frontend/backend contract drift (audit #9851) #9862 transcriber removal (my repoint was moot).
Post-rebase duplication 1.0405% < 1.05 guard threshold (won't false-fail).
duplication-guard.yml: verified --threshold 1.05 → exit 0, 0.5 → exit 1; YAML valid.

Model Used

Claude Opus 4.8 (1M context)

🤖 Generated with Claude Code

… dup lines (#9794) The extensions package was renamed to middleware (#7426) but only __init__.py and three builtin/ modules were converted to re-export shims; base.py, hooks.py, manager.py, hook_invoker.py, extension_manifest.py and builtin/__init__.py were left as full byte-identical copies (only the extensions.→middleware. import prefix differed). jscpd flagged this as the largest duplication cluster (~1500 lines across base/manager/hook_invoker/hooks). - Move transcriber_extension.py to canonical middleware/builtin/ (it had been added only to the deprecated extensions/builtin/ via #9044); export it from middleware/builtin/__init__.py; leave a re-export shim (preserves module-level router consumed by feature_routers via getattr). - Convert extensions/{base,hooks,manager,hook_invoker,extension_manifest}.py and extensions/builtin/__init__.py to thin re-export shims of middleware. - Repoint prod transcriber wiring (feature_routers.py) at the canonical module. - hook_invoker_test: patch middleware.hook_invoker.logger (where the shimmed implementation now logs) instead of the deprecated module symbol. Behavior-preserving: all import paths (extensions.base.X, extensions.builtin.*, etc.) still resolve; symbol identities preserved across the shim. Did NOT invent a new autobot_shared/hooks/ framework — there is one real subsystem (middleware) plus a backwards-compat alias; a third abstraction would couple, not simplify. Verification: extensions/ + middleware/ + transcriber test_extension = 120 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

gdrive.py and onedrive.py carried byte-identical copies of _content_hash, _extract_text_from_docx and _extract_text_from_pdf (jscpd flagged the docx/pdf pair as 19-line clones within the gdrive↔onedrive cluster). _content_hash was additionally duplicated in nextcloud.py and gitlab.py (functionally identical, only docstrings differed). - New knowledge/connectors/content_extraction.py holds content_hash, extract_text_from_docx, extract_text_from_pdf (verbatim from the identical copies). - gdrive/onedrive/nextcloud/gitlab import them (aliased to the existing private names so call sites are unchanged); drop now-unused hashlib/io imports. - OneDrive keeps its provider-specific xlsx/pptx extractors (genuine deltas). Deliberately NOT extracted: _classify_change and the OAuth _request helper — their bodies match but they depend on per-connector _load_ts/_store_ts that DIFFER between gdrive and onedrive, so hoisting them to the base would risk behavior change. Documented as a remaining cluster. Verification: gdrive_test + onedrive_test = 35 passed; connector batch suite 33 passed; import smoke confirms aliases resolve to the shared impl and the provider-specific extractors stay intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…#9794) autobot-backend/static/{js,assets}/index-<hash>.{js,css} are vite build outputs (outDir dist/, copied into the backend static dir on deploy), not source. Four hashed JS bundles were tracked though index.html references only one — the other three are stale prior builds, which jscpd flagged as 100+ line duplicate clones (index-B-LiTFoM.js <-> index-DyeT1pr8.js etc.). The existing .gitignore entries (static/js/, static/assets/) are anchored to the repo root and never matched the autobot-backend/ subdir, so the artifacts kept being tracked. Added autobot-backend/static/{js,assets}/ patterns and git rm --cached the 7 stale bundles (kept on disk; regenerated each deploy). index.html, favicon.ico and error_messages.yaml stay tracked (source / served fallback), matching the existing gitignore intent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…9794) Runs jscpd (cpd 5.x) over autobot-backend + autobot-frontend/src with the exact flags from the 2026-06-10 dedup sweep and fails when duplication exceeds the documented baseline (post-cleanup 1.04%; THRESHOLD 1.05 to absorb version jitter). Threshold-gate, not zero-duplication: known deferred clusters (#9856, #9859) need their own PRs; this only prevents growth. Ratchet THRESHOLD down as cleanup PRs land. Models the frontend-typecheck-regression guard; runs on PRs touching py/ts/js/vue and on push to Dev_new_gui; no untrusted event data in run: steps. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…9794) Per-cluster lines removed, new shared modules, before/after duplication % (1.16% -> 1.04%, -1,800 dup lines), and the clusters deliberately skipped or deferred with reasons (#9856 broken security package, #9859 Vue campaign). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-06-10T09:08:11Z

✅ SSOT Configuration Compliance: Passing

🎉 No hardcoded values detected that have SSOT config equivalents!

Applied Black, isort, and autoflake to match code-quality checks. Triggered by workflow auto-fix-formatting.yml. Co-Authored-By: Paperclip <noreply@paperclip.ing>

mrveiss

Code Review — CodeReviewer

Verdict: ⏳ Waiting for CI — code looks excellent, waiting for checks to complete

Summary

Behavior-preserving duplication elimination that:

Completes the → rename (#7426) by converting 6 full-copy modules to thin shims
Extracts byte-identical content-extraction helpers into shared
Removes stale build artifacts from tracking
Adds CI duplication guard (1.05% threshold, ratchet-down only)

Verification

✅ Code quality:

Public APIs preserved exactly via shims/aliases
Clean extraction of shared utilities
Comprehensive documentation in DEDUP_REPORT.md
155 tests passed (extensions + middleware + transcriber + connectors)

✅ Changes are correct:

extensions/base.py → thin shim re-exporting from middleware.base
Content extraction helpers properly extracted to dedicated module
Build artifacts correctly excluded via gitignore updates
Duplication guard YAML is valid

CI Status

All checks currently QUEUED on self-hosted runner (expected per CLAUDE.md). Monitoring for completion before final merge approval.

No blocking issues found in code review. Will auto-merge once CI completes successfully.

mrveiss

Code Review — CodeReviewer

Verdict: ⏳ Waiting for CI — code looks excellent, waiting for checks to complete

Summary

Behavior-preserving duplication elimination that:

Completes the extensions/ → middleware/ rename (#7426) by converting 6 full-copy modules to thin shims
Extracts byte-identical content-extraction helpers into shared content_extraction.py
Removes stale build artifacts from tracking
Adds CI duplication guard (1.05% threshold, ratchet-down only)

Verification

✅ Code quality:

Public APIs preserved exactly via shims/aliases
Clean extraction of shared utilities
Comprehensive documentation in DEDUP_REPORT.md
155 tests passed (extensions + middleware + transcriber + connectors)

✅ Changes are correct:

extensions/base.py → thin shim re-exporting from middleware.base
Content extraction helpers properly extracted to dedicated module
Build artifacts correctly excluded via gitignore updates
Duplication guard YAML is valid

CI Status

All checks currently QUEUED on self-hosted runner (expected per CLAUDE.md). Monitoring for completion before final merge approval.

No blocking issues found in code review. Will auto-merge once CI completes successfully.

mrveiss · 2026-06-10T09:32:14Z

Code Review — CodeReviewer

Verdict: ✅ Ready to merge

CI Status

All required checks PASSED:

✅ smoke-test (required gate): SUCCESS
✅ code-quality: SUCCESS
✅ All other checks: SUCCESS
⚠️ hardened-smoke-test: FAILURE (non-blocking per CLAUDE.md)

Code Quality

Behavior-preserving duplication elimination verified:

Extensions → middleware shims correct
Content extraction helpers properly extracted
155 tests passed post-rebase
Duplication reduced from 1.16% → 1.04%

Proceeding with auto-merge.

mrveiss and others added 5 commits June 10, 2026 11:55

github-actions Bot and others added 2 commits June 10, 2026 09:10

Auto-fix: code formatting

e5d60b1

Applied Black, isort, and autoflake to match code-quality checks. Triggered by workflow auto-fix-formatting.yml. Co-Authored-By: Paperclip <noreply@paperclip.ing>

Merge branch 'Dev_new_gui' into dedup-extraction

4f53a45

mrveiss commented Jun 10, 2026

View reviewed changes

mrveiss merged commit df9c429 into Dev_new_gui Jun 10, 2026
20 of 21 checks passed

mrveiss deleted the dedup-extraction branch June 10, 2026 09:32

mrveiss mentioned this pull request Aug 1, 2026

Stale branches detected #11703

Closed

mrveiss mentioned this pull request Aug 25, 2026

bug(automation): the stale-branch and unlanded-work sweeps test ancestry, so every squash-merged branch is reported stranded #15036

Closed

5 tasks

Navigation Menu

Repository navigation

refactor(dedup): eliminate code duplication via behavior-preserving extraction (#9794) - #9865

refactor(dedup): eliminate code duplication via behavior-preserving extraction (#9794)#9865
mrveiss merged 7 commits into
Dev_new_guifrom
dedup-extraction

mrveiss commented Jun 10, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026 •

edited

Loading

Uh oh!

mrveiss left a comment

Uh oh!

mrveiss left a comment

Uh oh!

mrveiss commented Jun 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mrveiss commented Jun 10, 2026

Thinking Path

What Changed

Deliberately skipped / deferred (documented)

Verification

Model Used

Uh oh!

github-actions Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ SSOT Configuration Compliance: Passing

Uh oh!

mrveiss left a comment

Choose a reason for hiding this comment

Code Review — CodeReviewer

Summary

Verification

CI Status

Uh oh!

mrveiss left a comment

Choose a reason for hiding this comment

Code Review — CodeReviewer

Summary

Verification

CI Status

Uh oh!

mrveiss commented Jun 10, 2026

Code Review — CodeReviewer

CI Status

Code Quality

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Jun 10, 2026 •

edited

Loading