UDT Erasure: include retained library functions when simplifying custom types

[Ian Davis (idavis)]

Summary

The branch fixes bugs where explicitly retained library callables could:

• survive DCE without receiving UDT erasure;
• retain struct construction, field access, or UDT types;
• reference constructor type items that were subsequently deleted;
• fail post-erasure validation;
• produce invalid execution graphs;
• panic during code generation;
• return different values after compilation;
• lose transitive UDT dependencies across package boundaries;
• crash reachability traversal when given an invalid pin;
• mutate unrelated packages during seeded erasure; or
• allocate inconsistent output when erasure runs more than once.

The core fix is to treat pinned callables as additional roots for UDT erasure, validate those roots before traversal, and transform only the entry- and pin-reachable package closure.

This branch fixes bugs where callables explicitly retained for code generation were preserved by dead-code elimination but skipped by UDT erasure.

The affected callables are “pinned” roots: callable items that are not reachable from the package entry point but must remain available because a caller intends to compile or invoke them directly.

1. Pinned callables retaining unerased UDT operations

Previously, UDT erasure followed only entry-point reachability. Item dead-code elimination, however, followed both the entry point and explicitly pinned callables.

As a result, a pinned callable could survive DCE while its body still contained:

• Ty::Udt types;
• struct construction expressions;
• UDT constructor calls or values;
• named field reads;
• field assignments; or
• struct-copy expressions.

Later compiler stages expect those forms to have been erased. Retaining them could cause invariant failures, validation errors, panics, or incorrect code generation.

The branch adds pinned callables as roots for UDT erasure.

2. Pipeline reachability disagreement

The underlying bug was a disagreement between transformation passes:

• UDT erasure transformed entry-reachable packages;
• item DCE retained entry- and pin-reachable callables;
• execution-graph rebuilding processed the retained items; and
• downstream validation or code generation consumed those retained items.

This produced a mixed FIR store where entry-reachable code had the post-UDT representation but pinned code still used the pre-erasure representation.

The branch gives UDT erasure and item DCE the same entry-plus-pins reachability roots.

3. Foreign pinned library callables being retained in an invalid form

A pinned callable may live in a library package that the application entry point never calls.

Previously, such a callable could be retained in the foreign package without that package receiving UDT erasure. Its body could then reach execution-graph rebuilding or code generation with unsupported UDT forms.

The branch expands the transformed package closure to include packages reached from pinned library callables.

4. Transitive UDT dependencies of pinned callables being skipped

A pinned callable can call another library function that constructs or returns a UDT:

operation Pinned() : Int {
    let data = Inner.Make();
    data.First * 10 + data.Second
}

Even if Pinned itself is retained, transforming only its immediate package is insufficient. The callable’s transitive dependencies may live in other packages and may also contain UDT types, constructors, struct expressions, or field accesses.

The branch computes reachability from the entry point and all pinned roots, then erases UDTs throughout the resulting package closure.

5. Struct-valued pinned callables failing after preservation

A pinned callable may construct a struct and read one of its fields:

struct Data {
    Value : Int
}

operation Pinned() : Int {
    let data = new Data { Value = 42 };
    data.Value
}

Previously, preserving Pinned could leave both the struct construction and field read unerased.

The branch ensures that pinned-only bodies receive the same struct-to-scalar-or-tuple lowering as entry-reachable bodies, preserving their runtime values.

6. First-class UDT constructors in pinned callables becoming invalid

Pinned callables can store UDT constructors as values or mix constructors with ordinary factories:

newtype Data = (Value : Int);

function Offset(value : Int) : Data {
    Data(value + 10)
}

operation Pinned() : Int {
    let factories = [Data, Offset];
    let data = factories[index](23);
    data::Value
}

If the pinned body is skipped by UDT erasure, its constructor value can continue referring to a type item that item DCE later removes.

The branch applies runtime-constructor replacement to pinned bodies, converting surviving constructor values into ordinary identity callables before type items are deleted.

7. Constructors and ordinary factories becoming indistinguishable

When a callable array contains both a UDT constructor and a user-defined factory, erasure must preserve their distinct behavior:

• the constructor returns its argument unchanged in the erased representation;
• the factory may modify the argument before constructing the value.

Skipping or inconsistently applying erasure to pinned bodies could invalidate the constructor or cause selection between the two callables to produce the wrong value.

The branch verifies both candidates remain distinct after the pinned callable passes through the full pipeline.

8. Pinned callable values changing during compilation

Because pinned bodies were not transformed consistently with their retained dependencies, compiling a pinned callable could change its observable result even though the original FIR evaluated correctly.

The branch adds semantic tests that evaluate pinned callables both:

1. before the pipeline; and
2. after the full transform pipeline.

This covers ordinary structs, constructor values, callable factories, and cross-package UDT dependencies.

9. Pinned items validated too late

Previously, pinned-item validation happened near item DCE and execution-graph rebuilding.

Once pins became roots for UDT erasure, invalid pins needed to be rejected before reachability traversal. Otherwise, a missing package or item could cause traversal to panic while attempting to expand the seeded graph.

The branch moves pin validation ahead of UDT erasure.

10. Missing pinned packages causing traversal panics

A pin can contain a package ID that does not exist in the store.

Passing that ID directly into seeded reachability could panic while looking up the package.

The branch validates the package and item before seed expansion and returns a structured MissingPinnedItem pipeline diagnostic instead.

11. Missing pinned items producing failures at the wrong stage

If an item was removed before being supplied as a pin, the pipeline previously detected the problem only at a later backend stage.

That allowed earlier transforms to mutate the store before reporting that the requested retained target did not exist.

The branch reports the missing pin before UDT erasure and before any pin-dependent traversal.

12. Non-callable pins reaching seeded erasure

Only callable items are valid pinned roots. A UDT type item or another non-callable item cannot serve as a directly compiled callable target.

The branch preserves explicit PinnedItemNotCallable validation and performs it before seeded UDT erasure begins.

13. Error ownership for invalid foreign pins

Diagnostics for invalid pins need to identify the package that owns the requested pin, including when the package itself is missing.

The branch keeps the pin’s package ID as the diagnostic owner, producing a useful structured error rather than an unrelated traversal failure.

14. Unrelated packages being transformed unnecessarily

A naive fix could run UDT erasure over every package in the store whenever pins are supplied.

That would:

• mutate unrelated libraries;
• allocate unnecessary FIR nodes;
• change package output that is outside the compilation;
• increase compiler work; and
• make the disposable codegen store’s scope less predictable.

The branch instead transforms only the package closure reachable from the entry point or the supplied pins. Tests verify that an unrelated package remains unchanged.

15. Seeded UDT erasure not being idempotent

Adding extra reachability roots must not cause repeated UDT erasure to generate additional constructor identities or rewrite already-erased pinned packages differently.

The branch verifies that running seed-expanded UDT erasure twice produces the same package representation after the first run.

16. Per-package ID allocation collisions in newly reached packages

Pinned roots can expand UDT erasure into packages that were not reachable from the entry point.

Any expressions, statements, patterns, blocks, or identity callables synthesized in those packages must use assigners initialized from the existing package contents.

The branch uses the pipeline’s per-package assigner registry for the expanded package closure, preventing generated IDs from colliding with existing FIR nodes.

17. Execution graphs being rebuilt from stale UDT bodies

Execution-graph rebuilding runs after structural transformations. If a pinned body was retained but skipped by UDT erasure, its rebuilt graph would describe unsupported pre-erasure expressions.

The branch erases UDTs in pin-reachable packages before DCE and execution-graph rebuilding, so the resulting graph corresponds to the transformed body.

18. Type items being removed while retained pinned bodies still reference them

Item DCE removes UDT type items after UDT erasure. This is safe only when every retained executable body has had its UDT constructor calls and constructor values replaced.

Previously, entry-reachable code satisfied that condition, but pinned-only code might not.

The branch ensures that all entry- or pin-reachable executable references are erased before type-item removal.