perf(codegen): stamp a pointer-free shape's typed layout into the allocation header

changelog.d/7834-alloc-band-typed-shape-bake.md

@@ -0,0 +1,86 @@
+### Allocation: the typed-shape layout is a property of the shape, so it is now a constant in the header
+
+Four allocation benchmarks sat inside a **6% band** at 2.50–2.65× node — object literals
+(`churn`, `churn_alloc`), class instances (`push_cls`), cyclic graphs (`cycles`). Four
+structurally different shapes do not land in a 6% band by coincidence; one shared
+per-allocation cost does.
+
+Symbolicated profiles of the 200M-allocation variants (two samples per program, agreeing
+within 1.5 pp) found it. `js_gc_declare_typed_shape_layout` was **30% of `churn_alloc` and
+`push_cls`**, and it spent that re-deriving *per object* a fact that is a property of the
+*shape*. #7510's memo had already collapsed the map round-trip to a direct-mapped probe;
+what remained was the probe, a type-table lookup, a field-count compare, and the cross-crate
+call itself. GC pause time, by contrast, is **3.5%** — the cost is that Perry performed nine
+out-of-line operations per allocation where V8 performs a bump-pointer and a write barrier.
+
+For a shape whose pointer mask is **statically empty**, the canonical layout is the constant
+`GC_LAYOUT_POINTER_FREE | GC_OBJ_TYPED_LAYOUT_INTACT`. The inline-bump `new` path already
+emits a packed `GcHeader` constant carrying the state half, so the intact bit is folded into
+that same store and the call is not emitted. What survives is the one half that depends on
+the recycled **address** rather than the shape — clearing whatever per-object record a
+previous tenant left — as a one-argument `js_gc_forget_object_layout` behind an inline
+`PERRY_PER_OBJECT_LAYOUTS_ANY` test. That global is a process-wide mirror of the per-thread
+emptiness flag, maintained by an armed-thread count; its `0` state proves every thread's
+per-object tables empty, and it is now also the first test inside `layout_forget_object`
+itself, replacing a Darwin `_tlv_get_addr` call with a static load on the disarmed path for
+every caller including object death.
+
+Two smaller levers in the same band:
+
+- **`js_ctor_return_override`** was called on every construction to answer a question that is
+  `undefined` for every constructor without an explicit `return` — 8% of `churn_alloc`, where
+  the synthesized object-literal constructor's only `ret` is the `TAG_UNDEFINED` constant.
+  `JSValue::is_undefined` is `bits == TAG_UNDEFINED`, so one 64-bit compare decides it inline.
+  The runtime call stays on the cold arm, where derived-constructor `TypeError`s, object
+  returns, arguments objects and arrays still need it.
+- **A `new` in a hot-loop callee is a `new` in a loop**, one frame out, so it now takes the
+  inline bump as well. `cycles.ts`'s `makeCycle` is the shape that needed this: 5 statements,
+  therefore `alwaysinline`, therefore *never* `inlinehint` — so the site gate was reading the
+  one flag that could not be set for the hottest function in the program. The signal is
+  `collect_hot_loop_callees` directly (≥1 in-loop call site AND ≤4 module-wide call sites),
+  which is the same anti-bloat bound the loop arm already accepts.
+
+Measured on the quiet M1 mini, best-of-5, with exit code 0 and byte-identical output verified
+for all 27 corpus programs before timing:
+
+| bench | before | after |
+|---|--:|--:|
+| `churn` | 0.4217 | **0.2900** (−31%) |
+| `churn_alloc` | 0.3720 | **0.2409** (−35%) |
+| `push_cls` | 0.3665 | **0.2368** (−35%) |
+
+`churn_alloc` goes **18.6 → 12.0 ns per allocation**; node is 7.1 ns on the same shape.
+Nothing else in the 19-benchmark corpus moves outside noise.
+
+`gc-handoff/bench/alloc_declare_pf.ts` and `alloc_declare_ptr.ts` isolate the cause with a
+control: the same program, same allocation count, same runtime stores, differing only in
+whether the second field's *declared type* makes the pointer mask non-empty. Before, both
+arms pay the declare and their times match (0.9044 / 0.8647); after, only the control does
+(0.5802 / 0.7845). Both move by the shared return-override lever (1.6 ns/alloc); the extra
+4.1 ns/alloc on the pointer-free arm is the layout declare itself.
+
+**Soundness.** The collector's view is bit-identical: `heap_payload_slot_selection` skips a
+`GC_LAYOUT_POINTER_FREE` payload without consulting any map, and the pre-existing path also
+reached `POINTER_FREE` for an empty pointer mask. A later pointer store still downgrades —
+with no descriptor to classify against, `layout_note_slot` falls through to its generic
+pointer-mask branch, which mints a per-object mask and flips the state to `SIDE_MASK`, a
+branch that needs no descriptor at all. A pointer-**bearing** shape keeps the full runtime
+declare and must: `SIDE_MASK` means the tracer reads a mask, and that call is what installs
+the shared `SHAPE_LAYOUTS` descriptor the mask lives in. (Installing it once at module init
+is not a substitute: the `keys_array` lives in the longlived arena and can be relocated by
+old-page defrag, and today's design survives that only by re-installing on the next
+construction.)
+
+One hypothesis was refuted rather than assumed: "INTACT set with no descriptor installed" is
+*not* a wrong-answer hazard. A JS number's NaN box **is** its double bits, so the raw-f64
+claim never changes the storage; every site that writes a slot raw re-proves the value finite
+inline; and every read that could treat those bits as a machine double is itself
+value-guarded. Verified directly — `(p as any).a = true` followed by `p.a + 1` through a
+warmed monomorphic accessor prints node's `2` on both arms, byte-identical.
+
+Tests: `crates/perry-codegen/src/lower_call/typed_shape_bake_tests.rs` — three IR-census
+ratchets of the "assert the subject was live" kind, one per direction (pointer-free bakes,
+pointer-bearing keeps the declare, `undefined` completion takes the inline arm). They earned
+that description: the first version **failed**, because codegen had scalar-replaced the
+probe's `new` and there was no allocation left to assert about. The escape is now explicit
+and commented, for the next person who writes a probe on this path.

crates/perry-codegen/src/codegen/function.rs

@@ -426,6 +426,12 @@ pub(super) fn compile_function(
     // functions (the hint would be redundant) and async/generator forms.
     // Try-containing functions are ordinary inline candidates since #7302
     // (invoke-EH removed the setjmp-era noinline requirement).
+    // #7834: record the raw admission, before the `inline_hint` window narrows
+    // it. `lower_call/new_alloc.rs` uses this to decide the inline-bump
+    // allocation, which wants "is this code hot" and not "may LLVM's inline
+    // threshold move" — an `alwaysinline` callee is excluded from the latter
+    // and is the hottest possible case for the former.
+    lf.hot_loop_callee = cross_module.hot_loop_callees.contains(&f.id);
     if !lf.force_inline
         && inline_hot_small_enabled()
         && (INLINE_HOT_SMALL_MIN..=inline_hot_small_size_cap()).contains(&f.body.len())

crates/perry-codegen/src/function.rs

@@ -36,6 +36,18 @@ pub struct LlFunction {
     /// inline-hot-small heuristic in `codegen/function.rs`. `alwaysinline`
     /// already implies the hint, so the two are never emitted together.
     pub inline_hint: bool,
+    /// #7834: `collectors::collect_hot_loop_callees` admitted this function —
+    /// it has at least one direct call site inside a LOOP and at most
+    /// `inline_hot_small_max_call_sites` call sites in the whole module.
+    ///
+    /// Not the same question as [`Self::inline_hint`], which is that set
+    /// INTERSECTED with a body-length window and with "not already
+    /// `alwaysinline`". A ≤8-statement function is `alwaysinline` and therefore
+    /// never hinted, yet it is exactly the shape whose body ends up executing
+    /// once per loop iteration — `cycles.ts`'s `makeCycle`. Sites that want
+    /// "is this code hot?" rather than "should LLVM's threshold move?" read
+    /// this.
+    pub hot_loop_callee: bool,
     /// Invoke-EH (#7302): this function contains landing pads (Itanium) or
     /// funclet pads (SEH), so its `define` line must carry
     /// `personality ptr @<name>` — `perry_eh_personality` on Mach-O/ELF,
@@ -215,6 +227,7 @@ impl LlFunction {
             linkage: String::new(),
             force_inline: false,
             inline_hint: false,
+            hot_loop_callee: false,
             personality: None,
             blocks: Vec::new(),
             block_counter: 0,

crates/perry-codegen/src/gc_call_effects.rs

@@ -83,6 +83,9 @@ pub(crate) fn classify_direct_callee(name: &str) -> GcCallEffect {
         | "js_gc_note_slot_layout_aware"
         | "js_gc_init_typed_shape_layout"
         | "js_gc_declare_typed_shape_layout"
+        // #7834: `layout_forget_object` behind a null check — two thread-local
+        // side-table removals, no allocation and no re-entry.
+        | "js_gc_forget_object_layout"
         // `typed_feedback.rs`: counters/registries only. This intentionally
         // does not include feedback wrappers that perform the actual object
         // get/set operation.

crates/perry-codegen/src/lower_call/mod.rs

@@ -71,6 +71,8 @@ mod scalar_method;
 /// module header for why the default build cannot fault on them.
 #[cfg(test)]
 mod timer_rooting_tests;
+#[cfg(test)]
+mod typed_shape_bake_tests;
 /// #7510: which of the two typed-shape layout entry points a `new` site emits,
 /// and where. Split out of `new.rs` to keep it under the 2000-line cap.
 mod typed_shape_init;

crates/perry-codegen/src/lower_call/new.rs

@@ -500,7 +500,8 @@ fn lower_new_impl_inner<'a>(
 
     // #7615 slice 8: the field-count computation and the three-arm instance
     // allocation moved verbatim to `new_alloc.rs` (see its header for why).
-    let obj_handle = super::new_alloc::emit_instance_alloc(ctx, class_name, class);
+    let alloc = super::new_alloc::emit_instance_alloc(ctx, class_name, class);
+    let obj_handle = alloc.handle;
     // #7154: root the instance for the duration of the constructor body.
     //
     // Until now the instance existed ONLY as an SSA register while that body
@@ -532,7 +533,7 @@ fn lower_new_impl_inner<'a>(
     //
     // Before the instance root's push, so the handle this names is the one the
     // allocator returned: nothing between here and there can collect.
-    emit_typed_shape_layout_declare(ctx, class_name, &obj_handle);
+    emit_typed_shape_layout_declare(ctx, class_name, &obj_handle, alloc.typed_layout_baked);
     let instance = {
         let protected = construction_runs_user_code(ctx, class_name);
         Instance {
@@ -669,16 +670,8 @@ fn lower_new_impl_inner<'a>(
                 || class.extends_name.is_some()
                 || class.native_extends.is_some()
                 || class.extends_expr.is_some();
-            let is_derived_lit = if is_derived { "1" } else { "0" };
-            let final_box = ctx.block().call(
-                DOUBLE,
-                "js_ctor_return_override",
-                &[
-                    (DOUBLE, &obj_box),
-                    (DOUBLE, &ctor_ret),
-                    (crate::types::I32, is_derived_lit),
-                ],
-            );
+            let final_box =
+                super::new_helpers::emit_ctor_return_override(ctx, &obj_box, &ctor_ret, is_derived);
             return Ok(final_box);
         }
         if let Some(save) = &saved_new_target {
@@ -1560,16 +1553,7 @@ fn lower_new_impl_inner<'a>(
         }
         ctx.current_block = after_idx;
         let raw = ctx.block().load(DOUBLE, &ret.result_slot);
-        let is_derived = if ret.is_derived { "1" } else { "0" };
-        ctx.block().call(
-            DOUBLE,
-            "js_ctor_return_override",
-            &[
-                (DOUBLE, &obj_box),
-                (DOUBLE, &raw),
-                (crate::types::I32, is_derived),
-            ],
-        )
+        super::new_helpers::emit_ctor_return_override(ctx, &obj_box, &raw, ret.is_derived)
     } else {
         obj_box
     };

crates/perry-codegen/src/lower_call/new_alloc.rs

@@ -42,9 +42,42 @@ use crate::types::{I32, I64, I8, PTR};
 /// scan-outward-past-switch-frames logic uses. A `new` inside a bare `switch`
 /// is therefore correctly treated as not-in-a-loop.
 fn new_site_is_in_loop(ctx: &FnCtx<'_>) -> bool {
-    ctx.loop_targets
+    if ctx
+        .loop_targets
         .iter()
         .any(|(continue_label, _, _)| !continue_label.is_empty())
+    {
+        return true;
+    }
+    // #7834: a `new` in a function the hot-loop-callee pre-pass admitted is a
+    // `new` in a loop, one frame out.
+    //
+    // The gate below this comment is about SPEED-vs-SIZE, and
+    // `collect_hot_loop_callees` answers exactly the question the loop test
+    // does — is this site hot enough to be worth ~268 bytes — with the
+    // anti-bloat backstop already attached: it admits only a function that
+    // (a) has a direct call site inside a loop and (b) has at most
+    // `inline_hot_small_max_call_sites` (4) direct call sites in the whole
+    // module. So the added code is bounded by 4 × (news in the function),
+    // which is the same order the loop arm already accepts.
+    //
+    // Deliberately NOT `func.inline_hint`: that is this set intersected with a
+    // 9..=20-statement window and with "not already `alwaysinline`", and the
+    // functions this needs most fall out of BOTH. `makeCycle` is 5 statements,
+    // so it is `alwaysinline` and never hinted — while being the single
+    // hottest function in the program.
+    //
+    // `cycles.ts` is the shape that needs it: `makeCycle` is called 10M times
+    // from `main`'s loop and allocates two `Cell`s, but its own body has no
+    // loop, so both allocations took the outlined
+    // `js_object_alloc_class_inline_keys` — 22% of the program's samples, plus
+    // a further 5% in `arena_alloc`'s inline-state sync, for work the inline
+    // bump does in eight stores.
+    //
+    // Reading `func.hot_loop_callee` here is well-ordered: `codegen/function.rs`
+    // sets it from `cross_module.hot_loop_callees` before the entry block is
+    // created and before any expression is lowered.
+    ctx.func.hot_loop_callee
 }
 
 /// Emit the instance allocation for `new <class_name>(...)` and return the raw
@@ -62,7 +95,37 @@ fn new_site_is_in_loop(ctx: &FnCtx<'_>) -> bool {
 /// emission, which is the `RootedGroup::adopt_emitted` push that roots it for
 /// the constructor body; nothing between the allocator call and that push can
 /// collect.
-pub(super) fn emit_instance_alloc(ctx: &mut FnCtx<'_>, class_name: &str, class: &Class) -> String {
+/// What [`emit_instance_alloc`] produced: the instance's user pointer, plus
+/// whether the allocation already stamped this class's canonical typed-shape
+/// layout into the object's `GcHeader` constant (#7834).
+pub(super) struct InstanceAlloc {
+    pub(super) handle: String,
+    /// `true` ⟹ the header already reads `GC_LAYOUT_POINTER_FREE |
+    /// GC_OBJ_TYPED_LAYOUT_INTACT`, so the construction site owes the runtime
+    /// only the address-dependent half of `js_gc_declare_typed_shape_layout`
+    /// (clearing a recycled address's stale per-object record).
+    pub(super) typed_layout_baked: bool,
+}
+
+pub(super) fn emit_instance_alloc(
+    ctx: &mut FnCtx<'_>,
+    class_name: &str,
+    class: &Class,
+) -> InstanceAlloc {
+    let mut typed_layout_baked = false;
+    let handle = emit_instance_alloc_inner(ctx, class_name, class, &mut typed_layout_baked);
+    InstanceAlloc {
+        handle,
+        typed_layout_baked,
+    }
+}
+
+fn emit_instance_alloc_inner(
+    ctx: &mut FnCtx<'_>,
+    class_name: &str,
+    class: &Class,
+    typed_layout_baked: &mut bool,
+) -> String {
     // Compute total field count including inherited parent fields.
     // The runtime allocates at least 8 inline slots regardless, so this
     // mostly matters for shapes >8 fields.
@@ -296,8 +359,41 @@ pub(super) fn emit_instance_alloc(ctx: &mut FnCtx<'_>, class_name: &str, class:
             // `js_gc_note_slot_layout` so the GC sees real pointer-bearing
             // slots regardless of this initial tag.
             const GC_LAYOUT_POINTER_FREE: u64 = 0x4000;
+            /// `GC_OBJ_TYPED_LAYOUT_INTACT` — the bit
+            /// `class_field_inline_guard` requires before it will read or write
+            /// a raw-f64 slot directly. Runtime-side name:
+            /// `gc::layout::GC_OBJ_TYPED_LAYOUT_INTACT`.
+            const GC_OBJ_TYPED_LAYOUT_INTACT: u64 = 0x1000;
             const OBJECT_TYPE_REGULAR: u64 = 1;
 
+            // #7834: when this class's canonical layout is declarable at
+            // allocation AND its pointer mask is statically empty, the state
+            // this header already carries (`GC_LAYOUT_POINTER_FREE`) is the
+            // FINAL one, and the only thing `js_gc_declare_typed_shape_layout`
+            // would add per instance is the intact bit. Stamping it into the
+            // same constant store removes the call: on `churn_alloc` /
+            // `push_cls` that call was ~30% of the program, almost all of it
+            // re-deriving per object a fact that is a property of the SHAPE
+            // (see `gc::shape_install`'s module docs — the memo already reduced
+            // the map round-trip to a direct-mapped probe, and what is left is
+            // that probe, the type-table lookup, and the call itself).
+            //
+            // Requires `field_count == slot_count`: that mismatch is the one
+            // case `init_typed_shape_layout` answers by DOWNGRADING
+            // (`layout_set_typed_unknown`), and a constant cannot express "it
+            // depends". Computed here, before `ctx.block()` takes its mutable
+            // borrow.
+            *typed_layout_baked = super::typed_shape_init::layout_pointer_free_at_allocation(
+                ctx,
+                class_name,
+                field_count,
+            );
+            let typed_intact_bits = if *typed_layout_baked {
+                GC_OBJ_TYPED_LAYOUT_INTACT
+            } else {
+                0
+            };
+
             let alloc_field_count = std::cmp::max(field_count as u64, MIN_FIELD_SLOTS);
             let payload_size = object_header_size + alloc_field_count * FIELD_SLOT_SIZE;
             // Round the whole allocation up to FIELD_SLOT_SIZE (8). The inline
@@ -410,7 +506,7 @@ pub(super) fn emit_instance_alloc(ctx: &mut FnCtx<'_>, class_name: &str, class:
             //   bits 32..63  = size (u32)
             let gc_packed: u64 = GC_TYPE_OBJECT
                 | (GC_FLAG_ARENA << 8)
-                | (GC_LAYOUT_POINTER_FREE << 16)
+                | ((GC_LAYOUT_POINTER_FREE | typed_intact_bits) << 16)
                 | ((total_size as u64) << 32);
             // GC_STORE_AUDIT(INIT): inline headers initialize freshly allocated unpublished object storage.
             blk.store(I64, &gc_packed.to_string(), &raw);