Skip to content

perf: avoid full-width post-clean narrative profiling - #181

Open
WilliamK112 wants to merge 1 commit into
FreshCode-Org:mainfrom
WilliamK112:agent/reuse-explain-context-cache
Open

perf: avoid full-width post-clean narrative profiling#181
WilliamK112 wants to merge 1 commit into
FreshCode-Org:mainfrom
WilliamK112:agent/reuse-explain-context-cache

Conversation

@WilliamK112

Copy link
Copy Markdown
Contributor

Summary

  • restrict explain_clean()'s post-clean context build to columns that can actually contribute a narrative
  • skip the post-clean context build entirely when there are no rationale-bearing actions or remaining missing values
  • retain full profiling for the pre-clean role table and the pipeline's separate post-representation engine cache, avoiding stale context reuse
  • add a 120-column regression test and an Unreleased changelog entry

Why

explain_clean() unconditionally profiled every cleaned column even though _narratives() only reads contexts for columns with rationale-bearing actions or for columns that remain missing. On a wide no-op frame, that doubled per-column context construction from 120 to 240 calls. Reusing the pipeline cache directly would be unsafe because it is built after representation repair and only covers engine-visible columns, while the role table describes the original frame.

The new selection preserves the existing narrative semantics while avoiding the redundant full-width pass.

Closes #32

Validation

  • red-first regression: the new wide-frame test failed before the implementation with 240 context builds instead of 120
  • pytest -m "not online and not large" — 4,052 passed, 6 skipped, 12 deselected; 93.16% coverage
  • pytest --no-cov -q tests/test_explain.py — 14 passed
  • ruff check src tests
  • mypy src/freshdata — no issues in 201 source files
  • git diff --check

AI assistance

I used OpenAI Codex to inspect the current profiling flow, implement the optimization, and run validation. I reviewed the final diff and test evidence before submission.

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Important

Review available on request

  • 🔍 Trigger review

Reviews should be triggered manually for repositories with fewer than 10 stars. Select Trigger review above or comment @coderabbitai review to review the latest changes. For a full review, comment @coderabbitai full review.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ba5d7352-9498-4518-8269-76d602a63669


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@WilliamK112
WilliamK112 marked this pull request as ready for review August 14, 2026 19:09
@strix-security

Copy link
Copy Markdown
Contributor

Strix is installed on this repository, but we couldn't run this PR security review because this workspace's trial has ended. Add a card to resume code reviews here.

So far, Strix has reviewed 36 pull requests, surfaced 3 security issues (1 critical/high) and blocked 1 risky merge across this workspace.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

explain_clean computes build_contexts twice — once on df and once on cleaned

1 participant