Skip to content

rust mode: long-lived module state has no retention bound (session maps never evicted, mc_changefeed never pruned) #521

Description

@iceteaSA

Refs are against master 62bf58c0c. These three surfaced in a read-only source audit, each confirmed at the cited lines. None is urgent; together they are the ck-mc process's and store's only unbounded retention I found. One shared module serves every Rust-mode session on the host and runs for days between restarts.

1. transform_session_roots is insert-only

crates/mc-module/src/lib.rs:3628 declares transform_session_roots: Mutex<HashMap<String, HashSet<PathBuf>>>. Transform binds add a per-session entry (:9501-9506, plus :5091/:5115). No production path removes, retains or clears it: grepping production code above the test module for removal on this map finds 0 sites. Route unbind and session.delete leave the entry in place, so resident memory grows with every distinct session the module has ever transformed. This is the one that matters.

2. guidance_dates can outlive a deleted session (narrow)

An entry is inserted when a session has no persisted guidance date yet (:9058-9062). It's removed once a later read finds the date persisted (:9051-9054), or when a transform response commits (:9998-10004). So it only leaks for a session deleted between its first guidance read and its first committed transform. That's a small window, and session.delete / route teardown don't remove the entry. It's minor; I'm listing it because it's the same fix site as 1.

Suggested fix for 1 and 2: evict per-session entries in the same place session.delete and route unbind already clean up. InFlight already has an LRU bound (MAX_IN_FLIGHT_SNAPSHOT_ENTRIES), so there's precedent.

3. mc_changefeed has no retention

crates/mc-store/src/lib.rs:987-996 and :1027-1138 define the changefeed and its appends, and :8111-8157 reads it. Nothing prunes it. It's small today (5,815 rows, feed_seq 1–5,815 on one host with one MODULE-authority project), but it's monotonic, and every Rust-owned memory or note mutation adds a row. Pruning safely needs a consumer watermark, because the host mirror consumes by feed_seq, so this is a small protocol addition rather than a blind DELETE.

Reachability: 1 and 3 grow on any host running rust mode, and 2 needs the narrow ordering above. All three are memory or disk growth over long uptimes, with no correctness impact.

Activity

  1. magic-alfonso commented on Sep 24, 2026

    @magic-alfonso

    Confirmed. transform_session_roots (lib.rs:3628) is written at 5091, 5115 and 9501 and never removed outside tests, so it grows with every session the module has transformed. We'll evict the per-session entries (and the guidance_dates one) where session.delete and route unbind already clean up. For mc_changefeed, agreed that pruning needs a consumer watermark rather than a blind delete, since the host mirror reads by feed_seq. We'll pick it up with the other Rust-mode fixes after the current release.

  2. magic-alfonso commented on Sep 28, 2026

    @magic-alfonso

    Status: parts 1 and 2 shipped in 0.43.2. The transform_session_roots and guidance_dates entries are removed when a session's last route closes, and on session.delete, and rebuilt from the store if the session comes back. Part 3, pruning mc_changefeed, isn't done yet. It still needs a consumer watermark so the host mirror can't miss rows. Keeping this open for that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions