Skip to content

pysimlin JSON converter silently drops Compat fields (data_source today; conveyor/queue/leakage/spreadflow/overflow) + ArrayedEquation.has_except_default #882

Description

@bpowers

Problem

pysimlin hand-maintains a mirror of the Rust json::Compat schema (src/simlin-engine/src/json.rs), and it is out of sync -- both the dataclass and the cattrs structure hook silently drop compat fields on every patch/edit round-trip.

The canonical Rust Compat (src/simlin-engine/src/json.rs:78) now has 10 fields:
active_initial, non_negative, can_be_module_input, is_public, data_source, and (new on the conveyor-engine branch) conveyor, leakage, spreadflow, queue, overflow.

pysimlin declares only 4:

  • src/pysimlin/simlin/json_types.py:35 -- the Compat dataclass declares only active_initial / non_negative / can_be_module_input / is_public. It is missing data_source (a pre-existing lag) and the five new conveyor / leakage / spreadflow / queue / overflow fields.
  • src/pysimlin/simlin/json_converter.py:490 -- structure_compat hand-reads only those same four fields, so it actively drops data_source today and would drop conveyor / leakage / spreadflow / queue / overflow.

The module header at src/pysimlin/simlin/json_types.py:2-3 claims "These types match the Rust JSON types in src/simlin-engine/src/json.rs" -- but they do not.

Separately, ArrayedEquation (src/pysimlin/simlin/json_types.py:55) is missing has_except_default, which exists in the Rust struct (json.rs:212) and in the TypeScript mirror (src/engine/src/json-types.ts:145 hasExceptDefault).

Why it matters (correctness -- active data loss)

Any pysimlin edit/patch that round-trips a variable through this converter demotes it:

  • A conveyor or queue stock loses its conveyor / queue marker and is silently demoted to a plain stock.
  • A conveyor leakage / spread / queue-overflow flow loses its leakage / spreadflow / overflow marker.
  • data_source (external-data reference) is dropped today, before the conveyor work even landed.

This is silent and irreversible on a routine edit. Same bug class as PR #810 / #811 (fidelity fields dropped on edit round-trips), but for pysimlin's Compat mirror rather than core's view/variable path.

Components affected

  • src/pysimlin/simlin/json_types.py -- the Compat dataclass (line 35) and ArrayedEquation (line 55).
  • src/pysimlin/simlin/json_converter.py -- structure_compat (line 490) plus the unstructure hooks and per-variable structure paths that build Compat (there are several Compat(...) / conv.structure(..., Compat) call sites around lines 544-704).

Suggested fix

  • Extend the Compat dataclass with data_source, conveyor, leakage, spreadflow, queue, overflow.
  • Add Conveyor / Leakage / SpreadFlow / Queue dataclasses mirroring json.rs (note SpreadFlow is adjacently tagged {type, distribution} in JSON; check the exact tagging against the Rust serde attributes).
  • Update the cattrs structure/unstructure hooks (structure_compat and the Compat-omit-when-default unstructure logic near line 132) to cover the full field set.
  • Add has_except_default to ArrayedEquation.
  • Add tests (the pysimlin gate is mypy strict + ruff + pytest); a JSON round-trip test asserting a conveyor/queue stock and leak/spread/overflow flow survive a structure/unstructure cycle would pin the regression.

Context

Discovered while fixing a json::Compat serialization drop on the conveyor-engine branch. The data_source drop is pre-existing (predates the conveyor work), and the five new fields are the conveyor/queue additions; both are follow-ups, not part of the conveyor-engine branch's regression fix. Distinct from #811 (core view-element fidelity) and tech-debt item #58 (TS Dimension type).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions