Fix cache_key collapse of non-pydantic private/__main__ objects

[ptomecek]

Summary

normalize_token's generic fallback in ccflow/utils/tokenize.py name-only-tokenized any object whose type module started with _ or contained ._, returning ("__internal__", module, qualname) and silently dropping instance state. Because "__main__".startswith("_") is True and private submodules (pkg._internal) contain ._, any non-pydantic object authored in a notebook/REPL (__main__) or a private module collapsed onto a single token.

Two genuinely-different values therefore produced the same cache_key, causing false cache hits / stale results — the dangerous direction (toward incorrect reuse rather than safe over-recompute).

class Add:
    def __init__(self, n): self.n = n
    def __call__(self, x): return x + self.n
Add.__module__ = "pkg._private"

compute_data_token(Add(5)) == compute_data_token(Add(7))  # was True (bug)

Note: pydantic BaseModels, functions, methods, code objects, partials and builtins all have dedicated handlers and were never affected — this only ever hit the fallback for arbitrary unregistered objects.

Fix

Make serializability the primary signal instead of the module name:

• Any object cloudpickle can serialize now folds its state into the token, so genuinely-different values stay distinct (fixes the reported collapse).
• The module-name heuristic is consulted only when serialization fails, to distinguish:
  • behavior-irrelevant interpreter/framework internals (e.g. _abc._abc_data, which Python 3.14 exposes in ABC-derived classes' function closures and which raise on cloudpickle.dumps) → stable name-only token, so behavior hashing does not crash; from
  • opaque user objects (including anything in __main__) → fail loud with TypeError rather than silently sharing a key.
• A small allowlist keeps picklable-but-volatile framework internals (pydantic compiled validators) name-only so their generated runtime state does not destabilize the token.

This is version-agnostic (3.11–3.14): it keys on the serializability property, not on an enumeration of internal modules.

Why this is the right shape

Instrumenting the old branch across the full test suite showed the only objects that ever reached it were the library's own internal helpers — including a stateful frozen dataclass that was itself being name-collapsed (a latent collision the fix also resolves). The ecosystem-internal (_abc/pydantic-C) case is forward-looking (Python 3.14). A static module allowlist would have been unmaintainable and would not even have covered the library's own modules; serializability is the durable signal.

Tests

New TestPrivateModuleNoStateCollapse covers:

• picklable instances in _secret / pkg._internal / __main__ → distinct and deterministic tokens;
• a stateful frozen dataclass in a private module → distinct;
• an unpicklable user object in __main__ → raises TypeError (no silent collision);
• an unpicklable interpreter-internal-looking object → stable name-only token;
• a picklable framework-internal (pydantic-core) object → name-only.

Validation

• Full test suite on Python 3.11: all passing (1336 passed, 2 skipped).
• Core normalize_token logic re-validated on Python 3.12.
• ruff clean.

[codecov]

Codecov Report

❌ Patch coverage is 98.03922% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.49%. Comparing base (6a788a6) to head (026456a).

Files with missing lines	Patch %	Lines
ccflow/tests/utils/test_tokenize.py	97.91%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #259      +/-   ##
==========================================
+ Coverage   93.48%   93.49%   +0.01%     
==========================================
  Files         176      176              
  Lines       20327    20377      +50     
  Branches     1350     1351       +1     
==========================================
+ Hits        19002    19051      +49     
- Misses       1052     1053       +1     
  Partials      273      273              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Test Results

    1 files  ± 0      1 suites  ±0   3m 5s ⏱️ +5s
1 338 tests +10  1 336 ✅ +10  2 💤 ±0  0 ❌ ±0 
1 344 runs  +10  1 342 ✅ +10  2 💤 ±0  0 ❌ ±0 

Results for commit 026456a. ± Comparison against base commit 6a788a6.

♻️ This comment has been updated with latest results.