Skip to content

Use effective identity for DryRunEvaluator node_key - #262

Draft
ptomecek wants to merge 1 commit into
mainfrom
fix/dry-run-effective-node-key
Draft

Use effective identity for DryRunEvaluator node_key#262
ptomecek wants to merge 1 commit into
mainfrom
fix/dry-run-effective-node-key

Conversation

@ptomecek

Copy link
Copy Markdown
Collaborator

Summary

DryRunEvaluator reports a node_key for each planned node. That key was derived from the raw structural cache_key(...) (effective=False), so it ignored the opt-in _CallableModel._evaluation_identity_payload() effective-identity hook.

Meanwhile, the in-process MemoryCacheEvaluator.key and the dependency-graph dedup (_build_dependency_graph) already key on cache_key(..., effective=True). As a result, a model that opts into a collision-safe effective identity was reported under the structural key in dry-run plans — inconsistent with how the same node is actually cached and deduplicated. For models whose distinct structural keys collapse, this could silently merge or mislabel distinct nodes in the plan/report.

Fix

Pass effective=True when building the dry-run node_key.

  • For models that opt out (the base hook returns None), this is a byte-for-byte no-op and preserves the existing structural key.
  • For opt-in models, the plan now reflects their effective identity, consistent with memory/graph dedup.

Test

Adds test_node_key_uses_effective_identity: a generated @Flow.model that ignores an unused ambient context field. The emitted node_key must equal cache_key(..., effective=True) and must merge two contexts that differ only in an ignored field — even though their structural keys differ.

Validation

ccflow/tests/evaluators/ and ccflow/tests/test_effective_key_characterization.py pass (75 tests), including all pre-existing dry-run tests.

The dry-run report node_key was derived from the raw structural
cache_key(...) (effective=False), so it ignored the opt-in
_CallableModel._evaluation_identity_payload() effective-identity hook.
The in-process MemoryCacheEvaluator and the dependency-graph dedup
already key on cache_key(..., effective=True), so a model that opts
into a collision-safe effective identity was reported under the
structural key in dry-run plans, inconsistently with how it is actually
cached and deduplicated.

Pass effective=True when building the dry-run node_key. For models that
opt out (the base hook returns None) this is a byte-for-byte no-op and
preserves the existing structural key; opt-in models now get their
effective identity in the plan, consistent with memory/graph dedup.

Add a regression test using a generated @Flow.model that ignores an
unused ambient context field: the emitted node_key must equal
cache_key(..., effective=True) and must merge two contexts that differ
only in an ignored field, even though their structural keys differ.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Signed-off-by: Pascal Tomecek <pascal.tomecek@cubistsystematic.com>
@github-actions

github-actions Bot commented Aug 23, 2026

Copy link
Copy Markdown
Contributor

Test Results

    1 files  ±0      1 suites  ±0   3m 17s ⏱️ +17s
1 329 tests +1  1 327 ✅ +1  2 💤 ±0  0 ❌ ±0 
1 335 runs  +1  1 333 ✅ +1  2 💤 ±0  0 ❌ ±0 

Results for commit b80b76a. ± Comparison against base commit 6a788a6.

♻️ This comment has been updated with latest results.

@codecov

codecov Bot commented Aug 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 95.23810% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.48%. Comparing base (6a788a6) to head (b80b76a).

Files with missing lines Patch % Lines
ccflow/tests/evaluators/test_reporting.py 95.00% 1 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #262   +/-   ##
=======================================
  Coverage   93.48%   93.48%           
=======================================
  Files         176      176           
  Lines       20327    20347   +20     
  Branches     1350     1350           
=======================================
+ Hits        19002    19021   +19     
- Misses       1052     1053    +1     
  Partials      273      273           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant