DECISIONS.md

DECISIONS

2026-09-22 — Qwen2.5-7B port of the four remaining read/write-feature studies (user decisions)

• Scope: results/qwen25_fv/ gets the four claims it lacked — FV (write-feature) ablation, read-feature ablation, read-injection → write-feature formation, FV presence vs accuracy — with the GPT-J methodology unchanged (same scripts, parameterised; only model, layers, dims, pool). Pool = the 96-task pruned split (77/19), FV = the 140-head ext_steerability_qwen25_96 selection.
• Read band for the task-unique direction: û_A = unit mean of the carrier-removed L11–13 label-token means (Qwen read-steering peak L12 ± 1; GPT-J used {5,6,7} around its L6 peak). The steering vector s_A = c + u_A uses the same band. Only the û_A rows are ported — the rank-1 raw m̂_A row and the attention-mask control of the GPT-J Appendix D table are NOT.
• No a-priori write / presence layer. The presence capture stores all 28 block outputs; the headline layer quoted in tables and figures (Studies B and D) is the layer where the 6-shot mean cos(cue residual, v̂_A) profile peaks, written to write_feature_and_model_accuracy/headline_layer.txt and stated wherever a number is quoted. The GPT-J band variants (max/mean over L9–20) stay for comparability. (The read→write ridge's best read layer, L20–22, is a read-side quantity and is NOT a write layer.)
• Counterfactual pairing: the 68 tasks shared with the 69-pool keep their GPT-J family tags; the 28 new tasks were tagged by a Haiku subagent with the make_cf_task_pairs.py prompt (the 8 tags Haiku assigned differently for shared tasks were discarded). Same seeded different-family sampling over the sorted 96-task list → artifacts/qwen25_fv/cf_task_pairs.json.
• Baselines: Qwen has no sixshot_dummy CSV, so the FV-ablation eval carries in-run, seed-matched zero_shot (0-demo prompt) and real{n}_baseline conditions; the read-ablation plots read those (same prompts, same T=1 readout) — the analogue of GPT-J copying its baselines from another run.
• Compute pattern (user request): a cheap orchestrator GPU pod on the shared volume runs every CPU-side step and dispatches sharded, resumable chains to a fleet of RTX PRO 4500 workers (logs/qwen25_fv/port/). Never touch pods this session did not create.

2026-09-03 — Task-unique part of the read feature = mean of the carrier-removed L5–7 residuals (user decision)

• For each ℓ ∈ {5,6,7}: $\hat c(\ell)$ = unit cross-task mean of $m_A(\ell)$ (shared carrier direction); $r_A(\ell) = m_A(\ell) - \langle m_A(\ell), \hat c(\ell)\rangle \hat c(\ell)$ (carrier PROJECTED OUT, not subtracted). $u_A = \tfrac13 \sum_\ell r_A(\ell)$ is the task-unique part — a vector with natural magnitude (median ‖u_A‖ 28 vs ‖c‖ 47).
• Ablation object: $\hat u_A = u_A/‖u_A‖$ (bankA/meanresid_top1_bases.pt). Steering object: $s_A = c + u_A$ with $c$ the layer-averaged L5–7 carrier (bankA/carrier_plus_meanresid_vectors.pt); no coefficient to fit.
• Supersedes the SVD definition (top singular direction $v_1$ of the three unit-normed residuals; $n_A = \langle \bar m_A - c, v_1\rangle$; $c + n_A v_1$): the two agree at |cos| median 0.9993, ‖u_A‖/|n_A| ≈ 1.001, and give identical ablation/steering numbers (±0.002). Reason for the switch: simpler to explain, no arbitrary SVD sign, natural magnitude. The SVD variant stays in App I as a labelled row; App F/K/presence prose notes the equivalence where their numbers were computed with $v_1$.
• 2026-09-03 (later): every downstream result that had been computed with the SVD $v_1$ (layer sweep, 6-shot by layer, $w_A$ variant, presence maps, Claim-4 cue effect, swap, App K cos) was re-run with $\hat u_A$ / $s_A$ and reproduces to ±0.005 — the paper reports only the mean-residual numbers (WORKLOG 2026-09-03). The swap's α is now in natural units ($\alpha u_A$, α=1 = natural magnitude), not $s_1$ units.
• Claim 6 (read→write map) is likewise stated on $u_A$: after train-mean centering the carrier drops out, so X = $u_A$ (task level and per prompt). Raw per-layer $m_A(\ell)$ maps stay only as the App G layer-readability sweep and one App J reference sentence.
• Notation in the draft: $u_A$ = task-unique part, $s_A$ = steering vector. (Before 2026-09-03, $u_A$ denoted the steering vector $c + n_A v_1$ — WORKLOG 2026-09-02 entries use that older meaning.)

2026-09-01 — Read feature estimator: final-target-site (bank a) is canonical (user decision)

• The read feature $m_A(\ell)$ is estimated from label_resid_means: the residual (block output) at the LAST token of the FINAL demonstration's target, averaged over the 150 clean 10-shot prompts. This is the paper's convention everywhere: ablation bases, steering vectors, regression inputs, presence maps.
• The ten-site-average estimator (label_avg10*, mean over all 10 demo targets) is retired to an appendix comparison (paper-draft Appendix K). It is a lower-noise estimate of the same feature (top-1 directions match at |cos| median 0.979) and gives sharper numbers where estimation precision matters (few-direction bases, ridge X) — quote it only as the estimator variant, never as the definition.
• Canonical artifacts: bases in bottom_up_ablation/bankA/, eval runs in bankA_* subdirs, per-prompt X in label_resid_perprompt/. Plot scripts take BANKA=1 to read them; the results/ deliverables are regenerated from bank (a) as of 2026-09-01.
• Headline ablation object (user decision 2026-09-01): the L5–7 top-1 task-unique direction (bankA/L5to7_top1_bases.pt) — the same $v_1$ the composite steering vector boosts, so necessity and sufficiency are stated for one direction. The 11-dir/top-3/top-1 (L5–15) and L6–9 variants stay in the App F ladder as comparisons.
• Headline steering object (user decision 2026-09-02): $u_A = c + n_A v_1$ — the task-agnostic L5–7 carrier plus the same ablated $v_1$ at its natural coordinate (bankA/carrier_plus_top1_vectors.pt, build_carrier_plus_top1_vectors.py); injected at L0 (6-shot 0.570 @α=2, 0.596 best). Full mean, mean-free, swap, $w_A$ = App I variants.
• Not yet re-run under (a): App G's detailed ridge sub-analyses (seed splits, centroid decomposition, oracle) and App I's carrier-gap hypothesis tests — marked with convention notes in the draft.

2026-08-31 — Paper-draft vocabulary: "target tokens", not "label tokens" (user decision)

• The paper draft (write_up/read_and_write_features_for_in_context_learning_paper_draft.md) and every figure it embeds say target token(s) — never "label token(s)". Same for derived phrases: target slots, demo targets, pre-target, dummy-target scaffold, random-target scaffold, mixed-task mixed-target 10-shot, gold target.
• Applies to visible plot text (titles, legends, axis labels, annotations). NOT renamed: artifact/results paths and setting keys (label_resid_means, labeltoken_fv_ridge, sixshot_randomlabel, test_mixedtask10, …), task names (pos_label, verb_tense_label), and code variable names.
• Figure scripts updated + figures regenerated 2026-08-31 (see WORKLOG). The circuit SVG (write_up/graphics/icl_read_write_circuit.svg) also uses "target".

2026-08-29 — Qwen2.5-7B-Instruct FV derivation → results/qwen25_fv/ (user decisions)

• New top-level bucket results/qwen25_fv/ (QWEN25_FV_DIR): repeat of the GPT-J 69-task FV pipeline (competence screen → pooled sparse head selection → zs/shuf/mix steering eval → prune-refit) on Qwen2.5-7B-Instruct. Sibling of chat_template_transfer, not inside it.
• Prompt convention: base sampling, NO prefix. Qwen prompts start directly with Q: — no chat template, no <|endoftext|> literal (that token is Qwen's PAD; Qwen trains with no BOS). The GPT-J <|endoftext|>-prefix convention is now dispatched per model family via model_utils.use_bos_literal (GPT-2-family only) — GPT-J runs are bit-identical.
• λ grid extended to [0.002, 0.005, 0.01, 0.05, 0.2] (both GPT-J runs chose the boundary 0.005; superset keeps comparability).
• Prune threshold is user-adjudicated this time: after round 1 the zs_best distribution is shown to the user, who picks the threshold (GPT-J used auto zs_best < 0.4). Qwen-pruned tasks are recorded ONLY in the qwen25 split JSON + results README — dataset_files/ is not touched (the GPT-J pool stays intact).
• Qwen pool: 104/117 manifest tasks pass acc6 ≥ 0.30 (screen = the ext117 plain-arm artifacts, same locked protocol; adapter make_qwen25_acc6_csv.py). Split seed 42 → 83/21. Capped small tasks kept (next_in_group/next_in_period: 52 train prompts; capture assert relaxed to ≥40 + all-consumed).

2026-08-28 — New research branch: chat-template transfer → results/chat_template_transfer/ (user decision)

• New live branch, separate folder by explicit user decision: does ICL transfer when demos are chat turns (user = input, assistant = label) instead of Q:/A:? Results go in the new top-level results/chat_template_transfer/ (CHAT_TEMPLATE_TRANSFER_DIR in paths.py) — NEVER into results/69_task_run/ ("should not pollute" the settled line) and not exploratory (it is live, forward-looking work).

• Model: Qwen/Qwen2.5-7B-Instruct (bf16). No thinking mode exists for Qwen2.5. System-prompt handling is an experimental variable: arms chat_blank_system (explicit empty system message — Qwen2.5's template otherwise auto-inserts its default "You are Qwen, …") and chat_no_system (system block stripped entirely), plus a plain Q:/A: arm on the same model as the model-change control. Exact rendered chat strings user-confirmed 2026-08-28 (country-capital example); eval_chat_template_ext117.assert_template guards the rendering at run time.

• Protocol = the locked extended n-shot sweep (compute_extended_nshot_sampled.py, 2026-08-13): 117-task manifest pool (user chose the FULL pool, not the filtered 69), n=6 (user revised from the initial 10), 50 prompts/task with the identical sha256 sampling seeds, T=1.0 pure ancestral sampling, first-line/strip/exact-match metric. Main deliverable = ranked bar chart in the style of extended_tasks_nshot_sweep/nshot_bar_6shot.png.

• The mainstream research line is the 69-task read/write-feature study (write feature = 37-head FV, read feature = early-layer label-token mean). Mainstream results buckets: results/69_task_run/ and results/steering_vector_comparison/. write_up/ is entirely mainstream. Anything essential to 69_task_run work counts as mainstream regardless of where it lives (e.g. src/sandbox/ext_steerability/ — its compute backend — and src/eval_scripts/regress_activation_to_fv_fulldim_ridge.py, a ridge library the labeltoken-ridge scripts import).

• Retired directions moved to results/exploratory/ (git mv, history preserved): direction1_ambiguous/, direction2_label_geometry/, direction3_fv_formation/, general/. They did not pan out; kept for possible revisits, but NEVER build new results on them without explicit user promotion (same rule as sandbox). See results/exploratory/README.md.

• Their scripts moved to src/eval_scripts/exploratory/ (129 files): every script whose only results bucket was a moved one (by grep on the AMBIGUOUS_DIR/LABEL_GEOMETRY_DIR/ FV_FORMATION_DIR/GENERAL_DIR constants), plus artifacts-only compute scripts of those directions (paired captures, payload/preimage fits, magid plots) and the retired Qwen3 side line (run_qwen3_*, run_varicl_qwen3). Their sys.path bootstraps were bumped one level and eval_scripts.<module> imports of moved modules rewritten to eval_scripts.exploratory.<module>.

• results/sandbox/ and src/sandbox/ are unchanged (already quarantined by the 2026-08-06/08 decisions).

• paths.py is why this move was cheap: the four bucket constants were reparented under the new EXPLORATORY_ROOT; no runtime code hardcoded the old paths, so scripts on old branches pick up the new locations automatically after merging main. Docstring/help-text path mentions in src/ were sed-updated; WORKLOG/DECISIONS history intentionally keeps old paths (prepend exploratory/ when reading pre-2026-08-28 entries).

2026-08-19 — FV cue-token ablation protocol (user-adjudicated) + pod lessons

• FV ablation study design (user decisions): ablate the unit 37-head task-FV direction from the residual stream at the FINAL CUE token of 6-shot prompts, prefill only; ops = zero-projection AND mean-ablation; BOTH layer clamps L9–27 and L0–27 reported side by side; readout = T=1 sampled exact match (sixshot_dummy protocol); counterfactual control = the shared artifacts/69_task_run/bottom_up_ablation/cf_task_pairs.json (different semantic family, Haiku-assigned, used verbatim by both ablation studies).
• Mean-ablation reference at the cue token = grand_mean_cue6.pt: equal-task-weighted grand mean over all 69 tasks of cue-token block-input residuals of the same 6-shot prompts (per layer). Note this differs from the pc50 label-token grand mean (token-count weighted) and lives at a different site — don't mix them.
• LESSON — stored T=1 sampled baselines don't reproduce exactly across runs even with shared crc32 seeds (fp16 logit differences flip borderline samples; antonym real6 0.36 vs stored 0.32). When comparing ablation arms against a baseline, recompute the baseline in-run (--with_baseline) so all arms share the run's exact logit stream; use stored numbers only as floors/references.
• LESSON — GPT-J snapshot glob is unsafe: the volume HF cache now has a second, stale snapshot dir (f3f..., only a stray model.safetensors) that sorted(glob)[-1] picks over the complete 47e... one → "Unrecognized model" errors. Pass --model_dir (or pin the snapshot hash) in pod drivers.
• Generation-stage OOM guard for long-prompt tasks (ag_news) on 31 GiB cards: token_budget 11000 / batch_cap 16 (pc50 values) + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

2026-08-16 — CANONICAL extended-task pool = the 69 steerable tasks (user decision)

• All future train/test splits and downstream experiments on extended tasks use ONLY the 69-task working pool unless the user explicitly asks otherwise. The canonical split is task_splits/extended_steerable_69_prunedfail.json (seed 43, 55 train / 14 heldout). Do NOT build new splits from the old 90-task extended_steerable_90.json, and do NOT reintroduce the pruned tasks implicitly.
• Provenance: 138 generated tasks → 48 fail the 6-shot ≥ 0.30 competence filter (kept in dataset_files/extended_tasks/ as documentation, grey in the figures) → of the 90 competent tasks, 21 could not be steered by the pooled sparse-opt FV (zero-shot best-layer acc < 0.4). Those 21 are MOVED to dataset_files/extended_tasks/head_intensive_pruned_tasks/ (own manifest + README with per-task pooled-zs labels) and are out of the working pool — to be revisited later as their own study, never silently mixed back in.
• The main manifest.json (117 tasks) defines the working pool; the plotting scripts (plot_extended_nshot_{sampled,bar}.py, plot_extended_tasks_catalog.py) filter to it.
• Current pooled head selection on this pool: 37 heads (λ=0.005), from artifacts/sandbox/ext_steerability/prunedfail_seed43/pooled_sparse/selection.json; steers 69/69 tasks ≥ 0.4 zero-shot (train .748 / heldout .734 mean best-layer).

2026-08-12 — LESSON: wordfreq/lemminflect mechanical dataset draws aren't reproducible across environments; freeze the final curated word list

• While strictly recurating dataset_files/extended_tasks/countable_uncountable.json (audit found ~186 dual count/mass-sense nouns mislabeled countable, e.g. water, blood, energy, plus a non-noun once, plus 4 mislabeled uncountable — childhood, adulthood, yogurt, theft), reran the task's generator's build_uncountable()/build_countable() (which gate candidates on wordfreq.zipf_frequency) in this repo's python3.12 (wordfreq 3.1.1) and got a different total (1076, not the original 1000) than the file already on disk — several borderline zipf values evidently differ from whatever environment/library version produced the original file, flipping which candidates the mechanical filter drops.
• Fix applied: for countable_uncountable.py, froze the final, already-audited word lists as literal data (_resources/generators/_frozen_lists.py) and made main() emit those directly instead of recomputing the mechanical draw at runtime. The draw functions/candidate lists stay in the generator as documented provenance but are off main()'s execution path.
• Any other generation_method="knowledge" generator that gates on wordfreq.zipf_frequency or lemminflect.getInflection for its final output (not just as a one-time curation aid) has this same latent reproducibility risk. Once a word list has been manually audited/accepted, freeze it as literal data rather than trusting the mechanical filter to reproduce the exact same set on a future rerun in a different environment.

2026-08-12 — 7 task datasets EXPANDED in place; no-validation split flag; isolation_prompts sets

• Canonical dataset files overwritten (originals appended-to, backups kept): present-past 293→885, singular-plural 205→1034, prev/next_item 225→674 each (script-generated sequence families, mirrored/inverse-checked), person-sport 318→733, person-instrument 510→670, product-company 522→906, national_parks 451→454 (NPS inventory exhausted). All LLM-generated entries restricted to pre-2021 well-known entities; contested forms excluded. Backups: dataset_files/abstractive/<task>.pre_expansion_2026-08-12.json (original = byte-identical prefix of the new file). country-capital/country-currency deliberately unchanged (hard-limited inventories). Old FV/capture artifacts built from the smaller datasets are unaffected on disk but no longer reflect the full example pools.
• No-validation example split: load_dataset(..., merge_valid_into_train=True) merges the 9% valid split into train (train 79% / test 21%); TEST MEMBERSHIP IS BIT-IDENTICAL to the old two-stage split. Opt-in flag (default unchanged); returned dict has no 'valid' key so stale callers fail loudly. New isolation-methods work uses this split.
• dataset_files/isolation_prompts/<task>/ (29 tasks, generator dataset_files/generate/generate_isolation_prompts.py, seed 42, pools deduplicated by input): 150 fixed 10-shot train prompts (demos+query from TRAIN, correct labels, distinct queries) + 30 test prompts per setting {zeroshot, sametask_shuffled10, mixedtask10 (10 demos from 10 distinct other tasks, correct pairs, all 28 eligible)}; the SAME 30 test queries across settings (paired design). Queries from TEST, demos always from TRAIN. See write_up/isolation_methods_levers.md for the study context.

2026-08-08 — Sparse-optimization head selection + vanilla_sparse_opt23 are SANDBOX-only

• src/sandbox/sparse_head_selection/ implements Hu et al. 2025 (arXiv:2505.05145 §3.1) joint sparse head selection on GPT-J (73 heads at c>0.2, λ=0.01 by leave-one-task-out CV), and vanilla_sparse_opt23 = the SANDBOX FV definition built from the 23 heads with c > 0.8: UNWEIGHTED sum of varicl mean head outputs (canonical stage-2 construction, the learned coefficients pick heads only). FV set: artifacts/function_vectors/gpt-j/sandbox/vanilla_sparse_opt23/.

• Despite clearly beating canonical AIE selection on held-out tasks (test-task best-layer zero-shot mean 0.600 vs 0.400 for varicl top-40; LOTO train-task 0.421 vs 0.193), this is a SANDBOX trial: "function vectors" still means train_varicl_top40 (2026-07-10) and nothing may build on the sparse sets without explicit user promotion. Known weakness: loses on lexical-relation tasks (antonym, synonym).

• Method conventions fixed by the user: zero-shot queries from valid split (cap 100 / min 80, train top-up), loss = raw −log p(full label) over greedy contextualized label tokens, injection ONCE at the cue token (block-9 output), c clipped to [0,1], LOTO CV for λ.

• Lesson: HF gradient checkpointing breaks (CheckpointError, saved-tensor count mismatch) when a forward hook injects a grad-requiring tensor inside a checkpointed block — use micro-batch gradient accumulation instead (frozen weights ⇒ only post-injection layers store activations).## 2026-08-06 — Sparse-optimization head selection is SANDBOX-only (not a new head-set standard)

• src/sandbox/sparse_head_selection/ + artifacts/sandbox/sparse_head_selection/ implement Hu et al. 2025 (arXiv:2505.05145 §3.1) joint sparse head selection on GPT-J (73 heads, λ=0.01, LOTO-CV). Despite clearly beating the canonical AIE top-40 on held-out-task zero-shot steering (0.421 vs 0.193 mean acc @L9), this is a SANDBOX trial: "function vectors" still means train_varicl_top40 (2026-07-10 decision) and nothing may build on the sparse head set without explicit user promotion.

• Method conventions fixed by the user for any follow-up: zero-shot queries from the valid split (cap 100 / min 80 per task, top-up from train split), loss = raw −log p(full label) with greedy contextualized label tokens, injection ONCE at the cue token (block-9 output), λ by leave-one-task-out CV, c clipped to [0,1].

• Lesson: HF gradient checkpointing breaks (CheckpointError, saved-tensor count mismatch) when a grad-requiring tensor is injected by a forward hook inside a checkpointed block — use micro-batch gradient accumulation instead (only layers after the injection store activations when the model is frozen).

2026-08-04 — Payload-subspace REPLACEMENT steering conventions (part 12)

• Two-step subspace-replacement op (user-specified, ordering is definitional): to steer a source-task prompt toward a target task inside the two k=4 d_payload subspaces, FIRST set the source-basis coords to the target task's mean coords there (v += (c(tgt->src) − v@B_src^T)@B_src), THEN set the target-basis coords to alpha·c(tgt->tgt). Running the target write LAST guarantees that wherever the two subspaces overlap the final state matches the steer-toward task while the source projection is minimised. alpha scales ONLY the target-write coords.
• Steering targets = UNPAIRED 10-shot task means (last demo's label token, train-split capture prompts, capture_payload_switch_means.py) projected into the cached bases — the point of the study is that no paired-prompt Delta is needed. Projection of the mean equals the mean of the projections (linear), so storing mean coords is exact.
• Eval comparability rule: task-switch steering studies with logit readout must rebuild the steer_switch_logit.py test pools verbatim (same build_pools, seed, n_test) so the alpha=0 clean baseline doubles as a reproduction check against the stored old-study baseline (reproduced here to 0.001 on both syn<->ant directions).
• Single-layer edits (block L output, capture stack row L+1); site = demo label token. See WORKLOG part 12 for findings.

2026-07-30 — direction3_fv_formation has a fixed 6-folder taxonomy

• results/direction3_fv_formation/ is organized as: ablation/ (causal projection- ablation studies, split preimages/ vs attention_head_mechanisms/ by direction source), attention_head_analysis/ (observational head studies), activation_to_fv_ decoding/ (ridge/PCA/OLS/cosine readability studies, controls nested under their study), activation_geometry/, preimage_analysis/ (pre-image work not tied to an ablation setup). NEW outputs must go under the matching folder — do not create new top-level dirs there. Old->new path map: results/direction3_fv_formation/README.md; migration script logs/direction3_reorg/migrate.sh; WORKLOG 2026-07-30 part 8. Pre-reorg paths cited in older WORKLOG/DECISIONS entries resolve via the README table.

2026-07-28 — Pre-image SUBSPACE ablation conventions (SANDBOX PP-preimage part 3)

• Subspace definition (user-decided; leading direction REDEFINED 2026-07-29): ablation subspace = QR-orthonormalized span{m_t − g, top-k CENTERED PCs} of the task's 170 per-prompt-FV pre-images, k ∈ {0,2,3,4}; m_t = task-mean pre-image, g = grand mean over ALL 27 tasks' 4590 pre-images at that cell. QR is sequential so subspaces are nested and the first basis vector is always unit(m_t − g). Mean-replacement target = the subspace component of g (matched population). Banks: fit_preimage_pca_subspace_banks.py --mean_mode taskoffset (the default), fp64 SVD/QR on CPU (CUDA gesvdj rule stays moot).
• NEVER use a raw-space pre-image POINT mean as an ablation direction. Pre-image points are reconstructed as x_raw = (x_c + x̄)·std + μ, so their mean is dominated by the population-mean activation (measured cos(m_t, g) = 0.99, cross-task cos 0.97+ at early layers): its removal is task-GENERIC devastation with a pathological late-L profile, and counterfactual controls degenerate (~1.25× vs Stream W's 4–9×). Directions must be built from DIFFERENCES of points (offsets cancel: m_t − g = std·P⁺(ȳ_task − ȳ_all)) — the same reason Stream W's dx = std·dz target-inverse never contained μ. The first run used the raw mean; the user was not shown this consequence beforehand (they should have been — definitional/geometry choices are user-gated like metric choices, 2026-07-14) and mandated redefinition + in-place OVERWRITE of run-1 outputs (recoverable only from git history ≤ 064cf65).
• Zero-op vs mean-op, the recurring lesson: clamping a subspace containing the task-mean pre-image to ZERO is devastating but almost task-unspecific (cf arms ≈ same damage) — the mean pre-image carries a big all-task shared component, so proj-to-0 is an off-manifold intervention, not a task-content removal. Report the MEAN op as the interpretable one and keep zero-op results only next to their cf controls. (Same asymmetry as the sink/proj findings in the persona-vector ICL study.)
• Comparability gates for Stream W-derived studies: any sibling script must import build_prompts/make_chunks/build_cf_map from ablate_oneshot_preimage_logprob.py (never re-implement) and hard-gate on prompt-identity equality (query/answer/answer_token_id arrays) vs the stored Stream W npz; clean-log-p agreement is ADVISORY only (cross-GPU fp noise ~1.7e-2, same-GPU rule from 2026-07-16 applies).

Reusable conventions, project decisions, and open questions. Append; date entries. Resolved questions move from "Open" to "Decided" with the rationale.

---

2026-07-27 — CUDA SVD: always pass driver="gesvd" for spectra/inverses

• torch.linalg.svd on CUDA fp32 defaults to the Jacobi driver (gesvdj), which showed backend error ‖W−USVʰ‖/σ₁ ≈ 1e-3 on 4096×4096 ridge maps (torch 2.12+cu130, RTX PRO 4500) — enough to fail a 1e-3 self-consistency gate on truncated-SVD pseudo-inverses. driver="gesvd" reaches CPU-LAPACK-level accuracy (~1e-5) and was FASTER (1.2s vs 3.9s). TF32 was off; purely the SVD backend. Convention: any CUDA SVD feeding singular values, ranks, or (pseudo-)inverses must pass driver="gesvd" (CPU tensors take no driver arg). Diagnostic pattern kept in logs/sandbox_perprompt/preimages/diag_svd_driver.py.
• rank90 convention for truncated inversions: k = smallest k with cum σ²/Σσ² ≥ 0.90 — the diagnose_weight_spectrum.spectrum_stats definition (σ² energy, NOT the plain σ sum; user-adjudicated 2026-07-27).

2026-07-20 — INPUT-matched two-shot token-pair pairing (--pair_mode input)

• The two-shot token-pair steering script now supports two pairing semantics: label-matched (default, original study: pairs share demo labels + query, demo inputs differ by function — input2 carries the counterfactual) and input-matched (--pair_mode input: pairs share demo INPUTS + query, each function supplies its own labels — label1/label2 carry the counterfactual, input2 is clean). Requested by the project owner 2026-07-20.
• Input-mode eligibility: shared input words with differing outputs across the two functions and all four demo labels single-token (mirrors the label-mode single-token constraint); one pair per eligible input word, no cap (user choices 2026-07-20). ant/syn 987 pairs, digits 200.
• Output roots: twoshot_tokenpair_perpair_inputmatch_cos_heatmap/ and twoshot_tokenpair_perpair_inputmatch_cumclamp_cos_heatmap/ under direction2_label_geometry; summaries carry a pair_mode key. The label-matched dirs/behaviour are unchanged (regression: 90/90 smoke grids bit-identical to the pre-change script).
• Interpretation guard: in input-matched pairs, label1→{input2, prelabel2} dircos ≡ +1.0 by construction (full causal patch for reads before label2) — treat those cells as sanity checks, not findings.

---

2026-07-20 — DEFAULT NUMBER TASKS = prev_number_digits / next_number_digits

• From now on the number tasks are the DIGIT variants by default (prev_number_digits, next_number_digits); the word variants (prev_number/next_number) are used only when explicitly requested. Stated by the project owner 2026-07-20.
• Why: GPT-J executes the digit tasks near-perfectly (top-1 exact ≈ 1.0 from 1–2 shots vs judged 0.70/0.45 plateaus for antonym/synonym — see the by_nshots judged study), inputs and outputs are 100% single GPT-J tokens (0–250), and there is no leakage adjacency to the prev_item/next_item train tasks that confounds the word variants' held-out status.
• Applied 2026-07-20: --tasks defaults flipped in cosine_activation_to_fv_varicl_pertask.py and plot_pertask_r2_best_lines.py (also its --input_csv default → the ..._plus_number_digits ridge study). Existing word-task results stay in place (tracked results are never deleted) but are no longer the reference.

2026-07-20 — Per-task cosine-to-FV metric (Stream Y): raw, per-prompt mean, train+test pooled

• Metric per (task, icl_example_index, token_role, layer) cell: cosine(activation, that task's train_varicl_top40 FV) computed per prompt, then averaged over all 170 prompts (130 train + 40 test pooled). Raw cosine — no centering. All three choices (per-prompt mean rather than cosine-to-the-mean-activation; both splits; raw) confirmed by the user 2026-07-20 before compute.
• Differs from the 2026-06-15 pooled cosine study (cosine_activation_to_task_fv) in FV definition (that one used the task-specific gptj_fv vectors) and granularity (that one averaged over the 29-task manifest; this one is per task, the 4 tasks of the per-task ridge R² study). Don't compare levels across the two studies.
• Interpretation caveat (observed): raw-cosine ordering across tasks (word > number) is the REVERSE of ridge-R² decodability ordering (number > word) — cosine alignment and linear decodability are distinct notions; don't use one as a proxy for the other.
• plot_pertask_r2_best_lines.py is now metric-generic via --value_column/--ylabel/ --title_metric/--suptitle; defaults unchanged (R² plot regenerates byte-identical).

2026-07-10 — CANONICAL FV DEFINITION: "function vectors" = train_varicl_top40 (GPT-J)

• train_varicl_max4_top40 was a DEBUG test set (variable-ICL capped at 4 shots), not a canonical FV definition. It has been moved to artifacts/function_vectors/gpt-j/debug/train_varicl_max4_top40 (README in debug/). Unless a study explicitly says otherwise, "function vectors" means artifacts/function_vectors/gpt-j/train_varicl_top40 (variable 1–10-shot CIE, top-40 multitask heads). Stated by the project owner 2026-07-10.
• Consequences: everything fit against the max4 FVs is debug-only — the Stream S ridge maps + preimage banks (preimage_pairdiff/train_varicl_max4_top40, ..._tsvdk16/...), the Stream S cosine analyses, and the Stream W v1 ablation run. Their run_config.json files record the pre-move fv_root path (left as historical provenance; fit_tsvd_preimages_multicell.py has a debug/ fallback). Script --fv_root defaults that pointed at the max4 set now point at the debug/ path.
• Note the FV-definition spread in existing results: the full-dim ridge R² study (fulldim_ridge_activation_to_fv/) used train_selected (top-10 multitask heads); Stream S/W-v1 used the max4 debug set. New preimage/causal work should refit ridge maps against train_varicl_top40 targets rather than reusing those maps.

2026-07-10 — Stream W conventions: projection-ablation indexing + single-task TSVD banks

• Ablation/edit-layer indexing, pinned: the preimage banks are keyed by edit_layer = capture_layer − 1 ∈ 0..27; edit layer b means hooking the output of transformer.h.{b} (= capture entry b+1 of the 29-entry residual stack whose entry 0 is the embedding). A "start layer L, all downstream layers" ablation therefore gates a hook over model_config["layer_hook_names"] with if block_idx < L: return output and NEVER touches the embedding entry (no preimage exists for it; maps start at capture layer 1). Start-layer sweeps run L = 0..27.
• Left-padded batching is safe for GPT-J logit readout without position_ids correction: GPT-J uses rotary (relative) attention and pads are masked, so a uniform left-pad shift leaves real-token logits unchanged; verified in ablate_oneshot_preimage_logprob.py's built-in batched-vs-unbatched cross-check. Site positions must still be offset by the per-row pad amount.
• Single-task TSVD preimage banks (fit_tsvd_preimages_multicell.py --tasks ..., <cell>/preimages/{task}_tsvd_preimage_bank.pt): the rank-16 inverse is linear in its target, so per-task banks were validated against the existing pairdiff banks via tsvd(fv_A) − tsvd(fv_B) == tsvd(fv_A − fv_B) (rel err ~7e-6). The script's skip-guard is now per-requested-bank-file (the old per-cell diagnostics.json marker would have skipped existing cells entirely and could never add new banks incrementally).

2026-07-08 — Shuffled-label control validates the full-dim ridge R² (Stream V)

• The activation→FV ridge R² is real signal, not pipeline leakage. Permuting the train-task→FV assignment (task-level; test targets true; identical pipeline incl. LOO-task alpha CV; 3 seeds averaged) collapses test R² from median 0.346 / max 0.465 to median −0.021 / max 0.0000 (best "control" cell is layer 0, the trivial train-mean predictor; only 1.2% of cells are even microscopically > 0, worst per-seed max 0.016). Ridge responds by pinning alpha high → predicts the train-mean FV. Control outputs: fulldim_ridge_activation_to_fv_shuffled{,_seed0,1,2}/; worker flags --shuffle_train_labels --shuffle_seed.
• Row-level variant (--shuffle_mode row, 2026-07-09) is equally dead and even cleaner: permuting the row→FV assignment across all 3400 train rows (balanced, fixed per seed) gives seed-mean test R² median 0.0000 / max 0.0028, median test MSE = V to 5 decimals (fulldim_ridge_activation_to_fv_rowshuffled{,_seed0,1,2}/). Two mode differences worth remembering: (a) row mode kills TRAIN fit too (train_mse ≈ V_train — conflicting targets for near-identical rows), task mode can still memorize train; (b) row mode's median R² is ~0 rather than slightly negative — shuffled targets are balanced within every LOO-CV fold, so the CV finds the global-mean predictor exactly optimal.
• Gotcha fixed en route: artifacts/residual_activations/*/index.json shard paths are ABSOLUTE and predate the 2026-06-19 results→artifacts reorg; any loader following them must fall back to the split dir (done in regress_activation_to_fv_fulldim_ridge.py, commit 42044d9). Other scripts copying this loader pattern have the same latent issue.

2026-07-07 — Ridge-map pre-images: NEVER use the exact inverse; use rank-truncated pseudo-inverse

• The full-dim activation→FV ridge maps have true rank ≤ #train tasks (20) — the targets are 20 distinct FVs, so W = (ZᵀZ+αI)⁻¹ZᵀY inherits rank(Y). Singular values below ~rank 20 (cond(W) ~ 1e9) are float round-off, not structure. Consequences (Stream S diagnostics, diagnose_pairdiff_preimage_spectrum.py + preimage_diagnostics/):
  • The EXACT pre-image direction is float noise: |dz_exact|/|dz_damped| ~ 3e7–1e8, and rounding W to fp16 (1e-3 perturbation) reorients it to cos ≈ 0.001 with the fp32 version. Any result computed from an exact inverse of these maps measures numerics, not the model.
  • The honest inverse is the rank-k truncated pseudo-inverse (k ≈ 16, ≤ 20) — fit_tsvd_preimages_multicell.py, banks in artifacts/preimage_pairdiff_tsvdk16/. It beat every other direction on the digits pair (mean cos +0.73–0.76 @L4, sustained 0.4–0.66 at label tokens where raw fv_diff is ~0) and slightly beat the gamma-selected Tikhonov damped variant (soft version of the same cutoff).
  • torch.svd_lowrank (q = k+32, niter=8) matches the exact fp64 top-k SVD to ~4e-7 on these spectra — use it instead of full 4096² SVDs (seconds vs minutes).
• Inverting the k=16 PCA ridge instead does NOT work as well (peaks ≪ fv_diff): its FV-PC basis covers only 24–38% of held-out fv_diff targets, and its top-variance activation PCs discard the low-variance directions that carry early-layer task identity. Variance-chosen ≠ regression-chosen subspaces.

2026-07-04 — Pair-diff pre-images: invert the FV DIFFERENCE, not per-task pre-images (Stream S)

• When a direction is needed for a task-pair difference under the ridge maps, damp the inversion of the DIFFERENCE target fv_A − fv_B directly (fit_ridge_preimages_multicell.py, pairdiff_preimages/ banks). The Tikhonov-damped inverse is linear in the target only at FIXED gamma; subtracting two per-task damped pre-images mixes two different gammas (the norm-cap selection rule picks per-target) and injects damping artifacts into the difference direction. Exact (gamma=0) inversion commutes with subtraction, so only the damped variant cares.
• Unit-normalizing does NOT make exact vs damped pre-images interchangeable. cond(W)~1e9-1e10 means the exact inverse's direction (not just norm) is dominated by the smallest singular values — Stream S confirms it empirically: exact pre-image direction explains ~random-floor levels of the two-shot pair-diff variance while the damped direction explains up to 0.43 (uncentered, digits query_final L4).
• Reusable intermediates: artifacts/preimage_pairdiff/train_varicl_max4_top40/<role>_icl{k}/maps/ holds materialized 4096x4096 ridge maps (W_std fp16 + scaler) for 6 cells x 28 layers with the varicl-max4-top40 FV target — any future pre-image/decoding study at those cells can skip the ~25 s/layer refits. Same layout as Stream R's artifacts/preimage_steering/ maps (icl10 pre_label, train_selected_top40 target).
• Centered vs uncentered explained-variance for paired diffs diverge wildly (digits demo2_label L4: top-1-direction bound 0.85 uncentered vs 0.13 centered): the pair-diff mean is a huge shared component. Report both; say which one a claim uses.

2026-07-04 — FV .pt files are dicts (weights_only) + FV-projection-ablation design notes (Stream T)

• compute_function_vectors.py saves each FV as a DICT {'function_vector' [resid_dim], 'top_heads', 'n_top_heads'} — NOT a bare tensor. top_heads holds numpy int64 scalars, so on torch≥2.6 torch.load(p) fails (default weights_only=True). Load pattern: obj = torch.load(p, weights_only=False); fv = obj['function_vector'].reshape(-1).float().
• GPT-J task-specific FVs live in artifacts/gptj_fv/<task>/<task>_function_vector.pt (top-10, n_shots=10). The _digits variants didn't exist and were computed 2026-07-04 (mean-acts 100 trials + indirect-effect 25 trials ≈ 9 min/task on an RTX 4000 Ada, 20 GB).
• Logit readout without the full lm_head: run model.transformer(...) under the edit hook, then model.lm_head(last_hidden[:, -1, :]). GPTJModel already applies ln_f, so last_hidden_state is post-norm — applying lm_head to the qfinal slice gives correct next-token logits while avoiding the [B, seq, 50400] allocation.
• FV-projection-ablation ordering: when steering (add) and ablation (project-out u) target the SAME token (qfinal), apply steer-THEN-ablate within each layer's hook.
• Localize BOTH the layer and the token when asking whether a direction mediates an intervention (Stream T lesson). "Steer across all layers" was ambiguous and cost two reruns. Three progressively finer designs gave different answers: (1) add the steer vec at ALL 29 layers at once → swamps a single-direction ablation, retention≈1, looks like the FV doesn't mediate; (2) one layer at a time but all 3 steer tokens together → reveals mid-net peak + ~0.33–0.48 α=2 retention; (3) one layer AND one token at a time (the right design) → a separate steer(ℓ)/steer+ablate(ℓ) curve per site. Result at (3): steering works at every site (qfinal peaks mid-net L10–14 gain up to +7.9; demo label tokens peak earlier and weaker), and ablating F'⊥F at qfinal removes ~50–70 % of the localized gain at α=2 at ALL sites — including gain injected back at the demo label tokens. Retention rises with α (strong steering brute-forces other directions), so the gentle-α regime is the clean test. Lesson: an all-at-once intervention conflates "does this direction matter" with "can I overwhelm the readout"; sweep every axis you're claiming localization over, and confirm the sweep granularity with the user before spending GPU. (Supersedes the first pass's mediation≈0 conclusion.)
• Which FV-derived direction to ablate — F'−F beats F'⊥F as an anti-steering knockout (2026-07-07). steer_twoshot_fv_ablation_logitgap.py --ablate_variant {fperp,fdiff}: fperp removes normalize(F'−proj_F F') (target-FV-specific), fdiff removes normalize(F'−F) (raw FV difference). At α=2 both give similar peak retention (~0.28–0.45); the difference shows at high α — F'⊥F retention climbs to 0.79–0.91 (steering brute-forces past it) while F'−F stays 0.00–0.66. Reason: the mean-difference steer vector is a (target−source) activation difference, so it aligns with the FV difference F'−F; F'⊥F throws away the F-parallel component and is a strictly partial knockout. Prefer F'−F when the goal is to maximally cancel difference-of-means steering. (Cheap variant runs: rename the old outputs with a _<variant> tag + re-run only the new variant; steer/clean columns must match byte-for-byte across variants since they're ablation-independent — a free correctness check.)

2026-07-03 — 10-shot strip steering (Stream Q): lm_head OOM fix + long-prompt batch tuning

• When you only need hidden states, call model.transformer(...), NOT model(...). model(...) computes lm_head(hidden_states) → a [batch, seq, vocab≈50400] fp32 tensor (e.g. 1.25 GB at batch 64 × seq 97), which we never use (we read residual streams via baukit TraceDict). This is what OOM'd the 10-shot run at the unsteered pass. model.transformer(input_ids, attention_mask) runs the embedding + blocks (where the hooks live) and skips lm_head/final head — identical captured outputs, ~1.25 GB less, and faster. The 2-shot script used model(...) and got away with it only because its prompts were short.
• Batch vs the 24 GB 4090 for these 29-layer retain_output captures: batch 64 pins memory at the ~23.4 GB ceiling (a longer-prompt batch then OOMs); batch 48 sits at ~21.7 GB with margin — use 48. The 29-layer retain_output (holds every block's [B,seq,D] output each forward) is the memory AND time driver, so runs are slow (~80 s/grid) regardless of the short prompts. Set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.
• Kill remote runs by PID, not pkill -f <pattern> when the pattern also appears in the killing shell's own command line — pkill -f steer_tenshot_strip matches the ssh shell running it and self-terminates (SSH exits 255, though it does kill the target first). Safe form: ps -eo pid,args | grep '[s]teer_tenshot_strip_cos' | awk '{print $1}' | xargs -r kill -9 (the [s] regex trick prevents grep from matching itself).
• Unmatched-demo mean-difference steering (10-shot, no matched labels): steer_vec at a structural slot = difference of the two tasks' MEAN activations there. Result: the steerable signal lives almost entirely in demo LABEL tokens (input/pre-label ≈ 0), mid-layer→late-read, distributed across demo positions (not just the last). Only the read token (qfinal) is byte-matched, keeping baseline_cos valid.

2026-06-30 — Two-shot token-pair cosine heatmaps (Stream P): conventions + GPU-pod gotchas

• Steering ≠ patching ⇒ byte-identity is NOT required. The readout adds α·mean_pairs[tgt−src] to a residual stream; that direction is well-defined at any token. So the search space can include the demo-2 INPUT token (t2 input2) even though it differs across functions. BUT its steer vector then carries a lexical/content component (not pure function context; at layer 0 it is literally a mean token-embedding diff) and its READ baseline cos is < 1 (vs ≈1 at the 5 clean tokens). Keep it, but flag it (suptitle + summary note_input2); don't compare its magnitude naively against clean tokens.
• Structural asserts that hold exactly: lower-triangle k ≤ i ≡ 0 (a position-i edit at layer i reaches a later position only at read layers > i, even across different tokens); embedding column i=0 ≡ 0 for CLEAN source tokens (same token ⇒ identical transformer.drop output ⇒ steer_vec=0; GPT-J has no additive positional emb at the embedding, so position shifts don't matter). input2 is the documented exception (assert skipped for it).
• Token positions for the 6-token 2-shot search space come from the inlined selected_token_records (loops both demos, icl 1 & 2): label1=last_label_token@1, prelabel2=pre_label_token@2, label2=last_label_token@2, qfinal=last_prompt_token@None; the two INPUT tokens are prelabel2−1 and qfinal−1 (no named role). Assert strictly-increasing positions per prompt.
• Memory-lean pattern for ≤20 GB cards: keep captured acts [N,6,29,D] in fp16; compute steer_vec and baseline cos in a chunked loop over pairs (CB≈64) to avoid full-N float copies; convert only per-read-token slices to float in the inner loop. Fits gpt-j-6b + acts at ~18.5/20 GB. Full-precision copies OOM.
• GPU-pod gotchas (RunPod): (1) a fresh pod created with --networkVolumeId but the DEFAULT --volumePath mounts the shared volume at /runpod, NOT /workspace (the templated pods use /workspace). Find the repo at /runpod/function_vectors, HF cache /runpod/.cache/huggingface. (2) Other pods only trust the USER's SSH key; to reach a self-created pod, pass our pubkey via --env PUBLIC_KEY="$(cat ~/.ssh/runpod_gpu.pub)" --startSSH. (3) The project docker image is now the upgraded torch 2.8 / cu128 stack (runs on Ada AND Blackwell sm_120) — but matplotlib is ABI-broken against numpy 2.1.2; pip install -U matplotlib (→3.11) before plotting. Compute scripts (numpy+torch only) are unaffected.
• Plotting is CPU-only — the plot scripts just read saved .npy/.csv grids. Install numpy matplotlib on the editing (CPU) pod (python3 -m pip install numpy matplotlib; note the system has TWO pythons — /usr/bin/python3.10 is the one the scripts use, NOT the /usr/local/bin/pip python3.12) and iterate on figures locally — no GPU pod needed once the grids exist on the shared volume.
• Comparability across many sub-plots: when per-figure color scales span a wide dynamic range (here ~100× across token-pairs), per-figure scaling is misleading. Add (a) a single GLOBAL vmax across all grids, (b) a token×token "matrix of heatmaps" master figure (spatially arranged, one colorbar), and (c) a scalar peak-Δcos overview matrix (annotated). Keep per-pair per-scale figures for drill-down.

2026-06-29 — Attention KNOCKOUT (Stream O): monkeypatch GPTJAttention._attn, pre-softmax mask

• GPT-J (transformers 4.49.0) uses EAGER attention by default (no GPTJSdpaAttention class), so pre-softmax attention scores are directly editable. load_gpt_model_and_tokenizer already loads eager; assert model.config._attn_implementation == "eager".
• Knockout recipe: monkeypatch GPTJAttention._attn with a faithful copy of the 4.49.0 source, and right AFTER attn_weights = attn_weights + causal_mask and BEFORE softmax, set the target (query,key) entries to torch.finfo(attn_weights.dtype).min. After softmax the key's weight is 0 and the row renormalizes to sum 1 — "ablate by zeroing the score, attention still sums to 1". Per-row, all-heads indexing: attn_weights[rows[:,None], heads[None,:], q_idx[:,None], k_idx[:,j][:,None]] = mn.
• Threading per-row indices: stash (q_idx[B], k_idx[B,K]) on each model.transformer.h[l].attn._ko before the forward (helper set_knockout/clear_knockout); the patched _attn reads _ko (None = clean). No baukit needed — read logits from the model output. Editing the 4D attention_mask does NOT work for per-row key positions (it's shared across the batch); editing self.bias fails under left-padding (same for all rows). Monkeypatch is the way.
• Verify with one output_attentions=True forward: knocked-out key weight ≈ 0, edited rows sum to 1.
• Result: qfin reads the task directly from the LABEL tokens, not from the demo-2 pre-label (cutting the pre-label edge ≈ cutting a structural-token edge; cutting the label edges collapses the task).

2026-06-27 — Patch-onset layer (--patch_from_entry) + regime folders; all-layers check

• The two patching experiments take --patch_from_entry (default 6 = "L6 and above"; 0 = all entries incl. embedding). Output goes to a per-experiment regime subfolder <exp>/{L6_and_above,all_layers}/ ("all_layers" if pe==0 else f"L{pe}_and_above"); plotters take --regime. Implemented by rebinding the module global PATCH_FROM_ENTRY at the top of main() so the existing hooks/asserts pick it up.
• Patching the embedding (entry 0) is a no-op for byte-identical positions (labels, query, pre-label in the paired design) — so the label-follow result is unchanged between regimes. It is NOT a no-op for the demo INPUT tokens (they differ across base/target): patching entry 0 there swaps the literal input word, which is why the interval grid's demo2-input cells jump in all_layers. Keep this in mind when reading any "all layers" patch that touches positions whose tokens differ across the pair.
• Conclusion: the label-token findings (Streams M/N) are robust to patch onset; document the all_layers run as a completeness check, not a new result.

2026-06-26 — Isolating a token's DIRECT effect (Stream N): pin-everything-else-to-base

• "Open" set-patching does NOT isolate the patched tokens. Overwriting token positions ← target with nothing else frozen lets the patched info be relayed: later tokens attend to the patched positions (from entry 7+) and pass the signal on. To measure a token set's DIRECT contribution to the output, also pin every other non-output token to its base value at the patched entries (h[:, :-1, :] = base_full[:, :-1, e], then set the patched roles → target; leave the last/output column free). Then only the patched positions carry target and only the direct patched→output path remains. The gap (open recovery − isolated recovery) = the relayed share.
• Requires capturing the base full residual stack (all positions, patched entries) — keep it per-chunk (seq differs per chunk under left-pad); ~3GB at N=544 on the 4090, fits with batch 64 + expandable_segments:True. (Stream M only needed the 6 named positions; isolation needs all positions.)
• Result: label tokens recover 74% (words) / 95% (numbers) of the task switch even fully isolated → the labels drive the output largely directly; pre-label ≈ 0. See [[stream-m-interval-patch]] WORKLOG.

2026-06-26 — Six-token interval activation-PATCHING (Stream M): mechanics + gotchas

• Interval-patch mechanic (vs Δ-vector steering): run the base prompt and, at residual entries 6..28, overwrite (assign, not +=) token i ← the other prompt's activation and token j ← the base's own clean activation. i then carries the other function for L6+; j is pinned to original (blocks relay). Hooks fire in forward order, so a downstream token k first differs at the entry after the first patched entry → Δcos(k, entry 6) ≡ 0 (good cheap invariant; asserted <1e-3).
• Pinning the OUTPUT token (qfin) to original zeroes any logit effect exactly — the entire j=qfin column is +0.000. Expected, not a bug: the final-position residual is restored to baseline so logits are unchanged. Keep the column (it's the causal boundary), don't special-case it.
• Reuse, don't reinvent the trace plumbing: the exact 2-arg (output, layer_name) factory closure, tuple-vs-tensor dispatch (output[0] for transformer.h.*, bare tensor for transformer.drop), and left-pad per-row position table all carry over verbatim from steer_label_cos_heatmap.py. The ONLY change for patching is assign instead of add, and a per-row position vector (pos6[:,i]) so the hook can edit two positions at once.
• Positions are NOT fixed even when every word is single-token. In-context tokenization shifts the query block by ±1 token for some words (e.g. acceptable), so the "all single-token ⇒ fixed index" shortcut fails. Always compute positions per row and align by ROLE (the named 6 positions), never by absolute index — base and target prompts of a pair can also differ in length.
• retain_output over all 29 layers is the memory bottleneck, not the stored captures: it holds every position × every layer for the whole batch. batch 256 OOMs the 4090 (~23 GB); batch 64 + PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True is ample and still ~17 short forwards/task.
• Single-token-input filter is free for antonym/synonym: 100% of demo-input options for the 544 used labels and 100% of the shared query pool are single-token (verified), so requiring it drops 0 tuples. Demo-input-1 and demo-input-2 sample from the SAME distribution (o2i[L]); only the query comes from the narrower shared_in pool.

2026-06-25 — Layer×layer label→query-final cosine-shift heatmap; env + reuse gotchas

New deliverable src/eval_scripts/steer_label_cos_heatmap.py (Stream L): 29×29 (intervention-layer × read-layer) heatmap of how injecting α·steer_vec(i) at the demo label token pushes the source prompt's query-final activation toward the (unsteered) target's, measured as Δcos = cos(steered src_final, tgt_final) − cos(src_final, tgt_final), meaned over overlapping prompt pairs. Layer 0 = embedding (transformer.drop), 1–28 = block outputs. Reusable facts:

• The grid is structurally upper-triangular and the embedding row/col is exactly 0. A label-token edit reaches the query-final token only via later blocks' attention, so Δcos = 0 for read k ≤ intervene i (incl. diagonal) — use this as a free correctness check. Embedding diff is 0 because the label + query-final tokens are byte-identical across f1/f2 (the overlapping design), so baseline cos at layer 0 = 1 and steer_vec[0] = 0.
• baukit edit_output hooks MUST be exact 2-arg (output, layer_name) closures (already noted 2026-06-11). Hit again here: passing per-chunk state via extra default-kwargs made the hook silently no-op (Δcos = 0 everywhere). Fix = a factory returning a 2-arg closure that captures state via scope. Also handle BOTH output shapes: transformer.h.{l} returns a tuple (edit output[0]), but the embedding module transformer.drop returns a bare tensor (edit output).
• matplotlib 3.7.1 is ABI-incompatible with this box's numpy 2.1.2 (numpy.core.multiarray failed to import on import matplotlib). Fixed by pip install -U "matplotlib>=3.8" → 3.11.0. Any plotting script will fail to import until this is in place.
• Shared output dir → name summaries per task (<task_pair>_summary.json, not a single summary.json) so a second task's run doesn't clobber the first's metadata. Grids/PNGs/CSVs are already per-task-named.
• Finding: mid-layer intervention band (words L5–12 peak L7–9; digits earlier, L4–8) feeding read L16–18, dead by intervene ~L16 — mirrors the 1-D causal window. Digits steer ~2× words; α=2 > α=4 at the peak (the toward-target-cosine metric overshoots sooner than the logit-flip readout). Absolute shifts are small (baseline cos near ceiling ~0.98–0.99); the location in the grid is the signal.

---

2026-06-19 — Results layout: gitignored artifacts/ vs tracked, direction-bucketed results/

Convention (use for all new scripts): import paths from src/utils/paths.py; never hardcode results/.... Roots are REPO_ROOT-anchored and env-overridable (FV_ARTIFACTS_ROOT, FV_RESULTS_ROOT, FV_LOGS_ROOT):

• ARTIFACTS_ROOT (artifacts/, gitignored) — intermediate caches: activations, captures, function vectors, head selections, scratch. Write all recomputable .pt/.jsonl here.
• RESULTS_ROOT (results/, TRACKED) — study deliverables, in 5 bucket constants by research direction: AMBIGUOUS_DIR (d1), LABEL_GEOMETRY_DIR (d2), FV_FORMATION_DIR (d3), STEERING_COMPARISON_DIR, GENERAL_DIR. Put figures in <bucket>/<exp>/figures/ or <bucket>/figures/.
• LOGS_ROOT (logs/, gitignored) — run logs.

What's committed: under results/ only *.png/*.pdf + summary *.csv/*.json (binaries *.pt/*.npy/ *.npz/*.jsonl/*.log and raw dumps per_query.csv/raw.json/recreate_fig8/**/*.md are globally ignored). Exception — head selections ARE committed even though they live in artifacts/: the top-N head rankings are the output of the expensive CIE computation, so multitask_top_aie_heads{.pt,_metadata.json}, heads.pt, heads_metadata.json, fv_manifest*.json, selected_heads.json are re-included via .gitignore negations (so FVs can be rebuilt without recomputing CIE). Scratch dirs (artifacts/_*/**) are re-ignored.

Gotchas: (1) .gitignore has NO inline comments — pattern # note makes the comment part of the pattern; put comments on their own line. (2) To track files under an ignored dir, ignore by glob (artifacts/**) + re-include directories (!artifacts/**/) + re-include specific filenames, with the negations LAST so they beat earlier rules (incl. the global *.pt).

---

2026-06-19 — TWO-shot matched-label paired captures (Stream K): construction + 5-role schema

Extension of the 1-shot paired-difference design to two demos. Use this construction for any 2-shot paired study:

• Matched labels, distinct within a prompt: demos carry labels (L1,L2), both from the shared-output pool, with L1≠L2. The label SEQUENCE is identical across the two functions (f1 prompt and f2 prompt use the same (L1,L2) and the same query); only the demo INPUT tokens differ by function (antonym-of-L vs synonym-of-L; for digits next-input L−1 vs prev-input L+1). So at every demo-label / pre-label position the token is byte-identical across f1/f2 — pure contextualization, as in the 1-shot design.
• Enumeration: one tuple per shared-output label word w → L1=w, L2=random distinct label, query from shared-input pool minus {L1,L2,4 demo inputs}, deterministic per (seed,task_pair,w). Yields n_prompts ≈ |label pool| (≈544 ant/syn, ≈198 digits) — comparable to the 1-shot run.
• NO train/test split for activation gathering (full-pool capture). The split only matters where Δ is fit-then-applied (steering); geometry is descriptive. (User-confirmed.)
• 5 captured roles per prompt: demo1_prelabel, demo1_label, demo2_prelabel, demo2_label, query_final. The *_prelabel roles are the A: token before each demo label (pre_label_token @ icl_example_index 1/2); demo2_prelabel was the specifically-requested new position. selected_token_records numbers demos 1-based (icl_example_index 1,2) and handles 2 demos with no special-casing.
• Script: src/eval_scripts/capture_and_grade_twoshot_paired.py (sibling of the 1-shot version; reuses get_residual_stack/selected_token_records/flush_shard + word_pairs_to_prompt_data). Output is an INTERMEDIATE → artifacts/twoshot_paired_graded/<pair>/ (git-ignored ARTIFACTS_ROOT), NOT results/. First-token rank grading stamped per row.
• Finding: a second matched demo ~doubles first-token top-1 on word tasks (antonym 0.232→0.439, synonym 0.066→0.221) and cuts the 1-shot copy-the-query failure; digits at ceiling (0.98–0.99).

---

2026-06-19 — Logit-readout switch-steering + clean train/test split (preferred over sample+judge)

For task-switch steering where the gold answers are single tokens, prefer the logit readout over the sample+judge harness — it's exact, ~100× cheaper, and needs no OpenAI key.

• Metric: steer source→target, then in ONE forward read logit(target_gold) - logit(source_gold) at the query final token. α=0 = clean baseline (negative = prompt favors source answer); steering pushes it up across zero = the switch. Script: src/eval_scripts/steer_switch_logit.py, plot plot_switch_logit.py (shaded band = 95% CI of the mean over the n=100 test queries = mean ± 1.96·SEM/√n, NOT per-query spread).
• DIGIT numerals are all single GPT-J tokens (verified 0–250, with leading space) — vs word-form numbers where compound forms >20 are multi-token. So for number tasks use the digit datasets (dataset_files/abstractive/{next,prev}_number_digits.json, inputs 1–200) to get single-token labels/golds. Word next/prev dropped for single-token analyses.
• CLEAN TRAIN/TEST SPLIT (no leakage in either the ICL example or the final query): the two positions Δ is read at are the demo label token (Δ_label) and the query final token (Δ_final). So hold out a TEST set of 100 label-overlap (shared-output) words AND 100 input-overlap (shared-input, single-token-gold-both) words; estimate Δ ONLY from disjoint TRAIN words, and draw the 1-shot demo for test prompts only from non-test words too (so a test query can't sneak in via the demo slot). Use equal n_train (=100 prompt pairs) for every task pair so Δ quality is comparable.
• Derive Δ from our own forward passes on train pairs, NOT load_capture_diffs over the full-pool oneshot_paired_graded capture — the latter pools over the whole shared vocabulary and overlaps the eval queries (train-on-test). Only hold out the test query words from Δ (the positions read); excluding their golds too is unnecessary and decimates the small digit pool (collapsed Δ to 5 words in one run).
• No correctness filtering on Δ collection (deliberate): activations are averaged over all train pairs regardless of whether the model answers correctly — this diverges from the standard ICL-correct-filtered FV recipe (get_mean_head_activations) and likely leaves Δ noisier (1-shot competence is modest), but the user chose to keep it blind. Revisit if Δ looks weak.
• Finding: two distinct causal windows by injection site — demo-label peaks early (L4–8, dead by ~L14); final-prompt-token peaks later (L10–12) and reaches larger logit-diffs. All effects gone by ~L16–20.

---

2026-06-18 — Behavioral switch-steering harness + GPT-4 judge must be parallelized

src/eval_scripts/steer_switch_judge.py (Stream E) is the behavioral analog of the logit-level steer_label_to_query.py: inject sign·α·Δ_site(L) into a source-task 1-shot prompt, generate n=10 temp-1 samples, GPT-4-judge them for the target task, sweep layer×α → accuracy curves.

• Reuse, don't recompute: the per-site steering vectors come straight from the existing results/oneshot_paired_graded/<pair>/ captures via load_capture_diffs (source role → Δ_label, target role → Δ_final). No new capture. sign = +1 if target is f1 (Δ is f1−f2) else −1.
• Steering hook during generation: baukit edit_output, add the vector at a per-row position only on the prompt forward (output[0].shape[1] > 1); cached single-token steps are skipped so the perturbation propagates through the KV cache. Replicate prompts ×n_samples explicitly (prompt-major order) + left-pad; label-site idx = pad_len + label_pos, final-site idx = −1.
• α=0 baseline is layer/site-independent → generate & judge it once per direction, draw flat. Sanity: an unsteered source-task prompt judged for the target task scores LOW (antonym-demo → synonym ≈ 0.08), confirming headroom for steering to lift.
• GPT-4 judge MUST be concurrent. The stock sequential judge() in judge_oneshot_paired.py runs ~7 batches/min (~8s/request) → ~27h for this sweep's 11k batches. A ThreadPoolExecutor (judge_parallel, --judge_workers 40, order-preserving, with exp-backoff retry on transient HTTP/JSON errors) cuts it to ~40 min. Use this pattern for any large judge run.
• Throughput seen on the 24GB Blackwell: GPT-J generation ~140 samples/s (gen_batch=20 queries × n_samples=10 = 200 rows/generate call); the full 564k-sample sweep generated in ~50 min.
• OPENAI_API_KEY is NOT in this session's env or /proc/1/environ (unlike earlier streams). An interactive-shell export does NOT reach the agent's fresh shells. Workable route: write the key to a gitignored file (.openai_key, chmod 600) and set -a; . ./.openai_key; set +a before the run.

---

2026-06-18 — variable-ICL FV variant capped at max 4 demos (top-40); zero-shot steering cost is small

New train_varicl sibling that caps the per-prompt random ICL count at 1–4 demos (--max_shots 4) instead of 1–10, top-40 heads pooled over the 20 train tasks. Single behavioral change; isolated roots so the original max-10 artifacts are untouched. NEW scripts: src/eval_scripts/run_varicl_max4_pipeline.sh (build driver — handles the stage-2 gotcha: test-task activations computed to results/_varicl_testtasks_max4/ then copied into the main root before the 29-task FV build) and src/eval_scripts/evaluate_heldout_varicl_max4.py (steering eval + 3-series plot; reuses evaluate_fv/get_filter_set/summarize_results, reads the max-10 N=40 curve from results/heldout_varicl_nheads_sweep/<task>/nheads_sweep_by_layer.json and the task-specific curve from results/heldout_multitask_head_eval/<task>/comparison_summary.json — no recompute of those lines, no no-FV-baseline recompute). Artifact roots: results/multitask_aie_heads_varicl_max4/, results/function_vectors/gpt-j/train_varicl_max4_top40/ (29 FVs, n_top_heads=40), results/heldout_varicl_max4_top40/ (9 per-task PNGs + AGGREGATE + summaries).

FINDING: shrinking the extraction ICL window to ≤4 demos costs a little zero-shot steering (mean best-layer top-1 0.353 vs max-10's 0.400; task-specific 0.483) and ~nothing few-shot (10-shot-shuffled 0.777 vs 0.784). Same mid-layer (L6–12) causal window, max-4 a notch below max-10 throughout; no task collapses. So the variable-ICL FV is fairly robust to a shorter extraction window — context at inference time compensates almost entirely, and even zero-shot the FV is only modestly weaker. The top-40 pooled head set is the canonical varicl ranking ((9,14)(15,5)(8,1)…), so the shot-cap changes the mean activations (and thus FV magnitude/direction) more than the head selection.

PCA-ridge activation→FV heatmap (k_act=16, k_fv=16) on the max-4 FVs: reran regress_activation_to_fv_pca_ridge.py (×10 ICL shards) + merge_fulldim_ridge_results.py with the FV target re-pointed to train_varicl_max4_top40 (cached residual activations in results/residual_activations/ are FV-target-agnostic → only --fv_root/--output_dir change). Out: results/pca_ridge_activation_to_fv_varicl_max4_top40/. Same canonical mid-layer (L9–15) bowl; best cell icl09/pre @ L13 = 0.1596 (FV-PC floor 0.1433). Compare via gap-above-floor (the FV-target-agnostic regression-quality metric, since each FV set has its own 16-PC variance floor): train_selected 0.0155, varicl_max4 0.0162, varicl_top40 (max10) 0.0214 → the max-4 FVs are as regressable from the residual stream as train_selected and a touch tighter than the max-10 varicl FVs. Do NOT compare absolute test_mse across FV targets (different floors); use gap-above-floor.

2026-06-18 — Paired 1-shot captures carry extra correctness notions (n=5 temp=1 fraction + prob-gated)

The paired 1-shot activation rows (results/oneshot_paired_graded/{antonym_synonym, next_number_prev_number}/shard_*.pt + grading.json) now hold these per-prompt correctness labels, all GPT-4.1-judged with the strict same-word-is-false prompts:

• judge_top1 — greedy decode (original).
• frac_correct_temp1_n5 (float) + n_correct_temp1_n5 (int 0..5) — 5 temperature=1 samples (seed 42), each judged; the row stores the FRACTION correct (k/5). This REPLACED an earlier single-sample judge_top1_temp1 (one noisy Bernoulli draw) — the n=5 fraction is the robust signal; the single-sample field was dropped from all rows. sample_judge_oneshot_paired.py (draws via num_return_sequences, reuses the judge helpers).
• greedy_answer_prob (float) — model's probability of the greedy answer = product of greedy per-token (top-1) probs over the answer span (tokens before first "\n"); equals first-token prob for single-token answers, joint product for multi-token (numbers). NB it is the probability of the greedy path itself, NOT the gold answer's probability (the answer judged IS the greedy output).
• judge_top1_p50 (bool) = judge_top1 AND greedy_answer_prob ≥ 0.50. (Threshold walked 0.70 → 0.60 → 0.50; each step a free instant re-gate from the stored float, no model rerun — the payoff of saving the continuous prob. p70/p60 fields removed. To change again, re-gate from the float.)

Activations are decode-independent (same forward pass), so all decode-dependent labels (temp=1 fraction, greedy prob/gate) are computed retroactively and stamped by match key (function_task, output_word, query) — same scheme as the original judge_top1 tagging. The prob gate makes NO API calls (reuses greedy judge_top1).

Lesson / caveat: in 1-shot, open-ended word tasks (antonym/synonym) are rarely high-confidence (mean greedy answer prob ~0.17) so even a 0.50 gate keeps ≤19 prompts (synonym 0) — too few for a correct-vs-incorrect geometry contrast on those tasks; use the continuous greedy_answer_prob or frac_correct_temp1_n5 as covariates, or the number pair (gate keeps 47–98). Confidence ≠ correctness (synonym: 8 prompts ≥0.50 but 0 judge-correct; prev_number 58 ≥0.50, 47 correct). Under temp=1 sampling numbers drop vs greedy (greedy argmax was right; sampling injects errors) while synonym RISES (.143 → mean frac .229 — many valid non-argmax answers). any-correct ≫ all-correct (most prompts are sometimes but not always right), which is exactly the spread the single sample missed and the fraction captures.

2026-06-18 — magnitude/identity datasets DOUBLED with decimals (canonical files overwritten)

dataset_files/ambiguous/{magnitude,identity}.json were extended from 300 → 600 entries (300 integer + 300 decimal, ≤3 dp) by dataset_files/generate/add_decimals_magnitude_identity.py. Integer-only originals backed up at {magnitude,identity}.int_only.json (restore from these to revert). Decimal entries mirror the integer structure: +d overlap (mag=id=d), −d differentiator (mag=d, id=−d). Split is now overlap=300 / differentiator=300. Impact: any re-run of FV extraction / disambiguation evals on these tasks now includes decimals; previously-saved FV .pt files (e.g. gptj_fv_ambiguous_constrained*) are integer-derived and unchanged. When judging numeric-answer correctness, use numeric equality (float(a)==float(b)), NOT normalize_answer (it strips .). Decimal labels are multi-token in BOTH tasks (e.g. 97.96→[' 97','.','96']), so the first/last-label-token capture finally matters for magnitude too.

2026-06-16 — Qwen3-8B 3+1+1 disambiguation + CAVEAT: first-token top-k invalid for digit-splitting tokenizers

Ran the Stream-F 3+1+1 ambiguous eval for Qwen/Qwen3-8B-Base (eval_ambiguous_disambiguation.py already takes --model_name). CAVEAT: the token-level topk scoring (score_topk, first answer token via tok(" "+ans).input_ids[0]) is invalid for Qwen3 numeric tasks — Qwen3 tokenizes " 47" as [space, "4", "7"], so the first token is a bare space for every numeric answer → gold and partner both "match" → round/truncate/first_digit/last_digit/count_* spuriously read top1=partner@1=1.00. GPT-J merges " 47"→single token, so it's fine. Use the whole-answer wordtopk (beam) scoring for cross-model comparison (eval_wordtopk_3plus1plus1_qwen3.json). Result (whole-answer): Qwen3 mean top1 0.678 vs GPT-J 0.561, mean partner@1 0.170 vs 0.294 — Qwen3 disambiguates better and is less prior-bound (round 0.06→1.00, british 0.33→0.87, reverse 0.00→0.28), but the prior asymmetry persists (last_letter 0.10/partner 0.60, largest_city 0.11/partner 0.70). Lesson: any first-token metric is tokenizer-dependent; prefer whole-answer scoring across models.

2026-06-16 — BUG: ICLDataset int-indexing upcasts mixed-dtype rows (float input corrupts int gold)

ICLDataset.__getitem__(int) does self.raw_data.iloc[i].to_dict(), building a 1-row pandas Series. A Series is single-dtype, so a row with a float column (e.g. round/truncate inputs like 3.7) upcasts an int output gold 4 → 4.0. List-indexing (ds[[i]], orient='list') preserves per-column dtype and is safe. Symptom: round/truncate scored 0.00 because golds were '4.0'; normalize_answer then strips the '.' → '40', so the model's correct '4' never matched. Only these two tasks are affected (only float-input + int-output tasks); demos were unaffected (they use list-indexing). Fix in recreate_fig8.py: extract the query via ds[[idx]]. After fix: round 0.00→0.77, truncate 0.00→~1.00. Lesson: never grab a single ICLDataset row with int indexing when columns have mixed dtypes; use list-indexing or coerce ints before str().

2026-06-16 — GOTCHA: task-name collisions across dataset folders; key results by <folder>/<task>

4 task names exist in BOTH dataset_files/abstractive and dataset_files/ambiguous: count_consonants, count_vowels, identity, magnitude (the ambiguous copies are the overlap/differentiator-structured variants; load_dataset only scans abstractive+extractive and would silently grab the abstractive one). Any results store keyed by task name alone (results/.../<task>.json) will have the second folder's run clobber the first. Convention going forward: key per-folder results by <model>/<folder>/<task> (recreate_fig8.py does this).

2026-06-16 — Recreate Fig 8 for two models: generation-until-newline, not first-token rank

For the Fig 8 recreation (Stream H), accuracy is greedy generation stopped at "\n" then scored with normalized exact_match_score over the full answer — matching Stream G's correctness definition, NOT the cached first-token clean_topk. Efficient multi-shot readout: per trial sample 10 demos + 1 query, and for k=0..10 use the first k demos (nested prefixes of the 10-shot prompt) so shots 1..9 come from the same draw. Batched generation with token-budget batching (left-pad) keeps GPT-J's full-MHA KV cache under 80GB. All 2200 responses/task are saved to <task>_responses.jsonl for inspection. (A single-forward "read every demo answer position" trick is ~11× cheaper but only works for first-token scoring; generation requires real decoding, so we batch instead.)

2026-06-15 — Per-prompt correctness for the residual-activation captures (Stream G)

To mark capture prompts correct/incorrect, do not reuse fs_results_layer_sweep.json: it scores a different dataset partition (the 504-example valid split) than the captures (130 train + 40 test sampled by stable_rng(seed, task, split, "query_indices")), and stores only first-token ranks.

Instead, compute_capture_prompt_correctness.py regenerates the exact capture prompts by importing the capture's own deterministic helpers (sample_query_indices / sample_demo_indices / make_prompt from extract_targeted_residual_stream_activations.py, seed 42) and asserts the regenerated query_source_index order equals the capture index.json's query_indices (byte-identical prompts).

"Correct" = full-answer match, not first-token rank (per request): greedy generate (do_sample=False, max_new_tokens=16), then the repo's parse_generation(out, [target], exact_match_score) — its regex ([\w. ]+)[\nQ]* truncates at the newline / next Q:, and exact_match_score is SQuAD-normalized. Join correctness to activations by (split, query_source_index) (robust; verified zero misses). Correctness is one bit per prompt, shared across all 31 token positions of that prompt.

2026-06-15 — Cosine(activation, task-specific FV) heatmap (Stream G), companion to the MSE study

Same 31-token-position × 29-layer grid as the full-dim ridge MSE study, metric = per-example mean raw cosine (no centering) between each residual activation and that task's OWN task-specific FV (results/gptj_fv/<task>/...), averaged over all 29 tasks. cosine_activation_to_task_fv.py is model-free; reuses the ridge loader constants + the merge script's position_key/label axis ordering; diverging heatmap centered at 0, shared scale across the all/correct PNGs. Each activation directory targets one ICL index, so token_role alone uniquely buckets a prompt's rows (no icl filter needed). Result: all-positive, mid-layer bowl (peak L10), role banding last_prompt_token≈pre_label_token ≫ first/last_label_token; correct-only ≈ all at the aggregate level (max|Δ|=0.0016).

2026-06-14 — ambiguous task-disambiguation datasets + 3+1+1 eval; FINDING: prior-bias asymmetry

New dataset family dataset_files/ambiguous/ (generator create_ambiguous_datasets.py) for the task-DISAMBIGUATION study: pairs (f1,f2) that AGREE on an overlap region and DISAGREE on a differentiator region (matched input set; overlap entries byte-identical in both files). Four pairs: magnitude|identity (50/50), past_tense|past_participle (50/50), first_letter|last_letter (50/50), capital_city|largest_city (50 overlap / 35 differ — only ~35 real capital≠largest countries exist, ~20 GPT-J-plausible; recall caveat, trim/restrict downstream).

Eval eval_ambiguous_disambiguation.py (GPT-J, n=100/task, cross-prompt batched greedy gen, token-level exact match; matches_partner = model emitted the OTHER function's answer). Prompt = 3 overlap demos + 1 differentiator demo (task's output) + 1 differentiator query.

FINDING — a single disambiguating demo only redirects GPT-J in the prior-aligned direction. magnitude/identity are ~perfect both ways (.98/1.00). The other pairs are strongly asymmetric: the model nails the prior-aligned task and ignores the 4th demo for the anti-prior task, emitting the prior answer instead — past_tense .93 vs past_participle .36 (partner .60); first_letter .97 vs last_letter .04 (partner .89); capital .57 / largest .47 (noisy, recall-limited). So 3 ambiguous

• 1 disambiguating demos are enough only when the two functions are equally "natural"; a strong prior (past-tense over participle, first- over last-letter) is not overcome by one differentiating example. The matches_partner rate is the key diagnostic (separates "did the other rule" from "neither"). Next probes: a control arm (4th demo also ambiguous → prior baseline) and k>1 disambiguating demos.

2026-06-14 — train_selected FVs now also at top-20 and top-40 heads

The train-pooled (train_selected) FVs exist at three head counts: top-10 (original, results/gptj_fv_multitask_top10/), top-20 (..._top20/), top-40 (..._top40/), all 29 tasks. Built from the same results/multitask_aie_heads/multitask_top_aie_heads.pt (stores top-40) + cached mean activations — no CIE/forward recompute. Organized views: results/function_vectors/gpt-j/train_selected_top{20,40}/. FV norms grow with n (more out_proj terms summed): top-10 ~30–47, top-20 ~34–54, top-40 ~41–66. NB the n=20/40 head sets are strict prefixes of the top-40 ranking, so head sets nest (top-10 ⊂ top-20 ⊂ top-40). Relevant to the n=10 degeneracy note below: train vs train+test first diverge ~n=11, so the top-20/40 train_selected FVs are the right inputs for a non-degenerate train-vs-train+test comparison (rebuild train+test at matching n first).

Held-out steering finding (9 test tasks, evaluate_heldout_multitask_head_fvs.py --n_top_heads {10,20,40}; combined results/heldout_multitask_head_eval_nheads_comparison.json): adding heads helps ZERO-SHOT steering but not ICL-context steering. Best-layer zero-shot+FV mean top-1 rises 0.376 (n10) → 0.381 (n20) → 0.446 (n40), closing most of the gap to task-specific (0.483); some tasks jump a lot (capitalize 0.75→0.96, capitalize_first_letter 0.70→0.95). But 10-shot-shuffled+FV is flat/down (0.796→0.785→0.780; task-specific 0.812) — with real context present the extra heads add nothing. Caveat: at n40 several tasks' best zero-shot layer drifts to L0–1 (larger-norm FV), so verify those before citing; the mid-layer L8–13 optimum is stable for antonym/synonym/capitalize_first_letter.

2026-06-11 — FINDING (Phase 2): the mean label-token function axis is CAUSAL — steering it flips synonym→antonym

steer_label_to_query.py, results in results/oneshot_steering/. Injecting α·Δ_label (mean antonym−synonym label-token difference, computed on the 530 Phase-1 words) at the demo label token of a synonym-context 1-shot prompt, scored on 1,003 disjoint queries with single-token gold ant+syn:

• Geometric propagation: the induced query-token shift aligns with the natural syn→ant direction — cos(shift, Δ_final) ≈ 0.71–0.76 (steer L6, read L8; >0.5 through L15), matched-norm random control ≈ 0.01–0.06. α-stable over 0.5–4 (linear regime), degrades at 8.
• Behavioral flip: flip rate 0.283 (clean) → 0.64 (L6, α=8); mean logit(ant)−logit(syn) −1.44 → +1.12, zero-crossing at α≈2. Random control stays negative (−0.92 @ α=8) — effect is direction-specific. Best steer layers 6≈9 > 11.
• Reconciles Phase 1's negative map result: the dominant shared axis propagates causally and predictably label→query; what the (weak, high-rank) linear map could not capture is the per-word high-dim tail. So "label-token arithmetic moves the next position predictably" is TRUE for the function axis specifically, FALSE for the full per-word geometry.
• Geography (landmark↔park) propagates weakly (cos 0.13–0.23, n=84, geometry-only — no behavioral flip exists since both functions output the same country).
• Causal window [L4, ~L15] (full L1–L24 sweep): steering is null at L1–2 (flip ≈ baseline despite maximal downstream depth; natural ‖Δ_label‖ ~0.7–0.9, the axis doesn't exist yet), onsets sharply L3→L4 (flip 0.34→0.57 @α4), plateaus L4–9 (peak cos 0.76, flip 0.60), decays L11–12, and is exactly baseline-dead by L16 (L16/20/24: flip 0.282–0.284, cos ≤0.09). Interpretation: the function fingerprint is written into the label token ~L3–6 and transported to the query position before ~L16 — congruent with the CIE top-head band (L8–15). Profile: results/oneshot_steering/fig_steer_layer_profile_L1to24.png.
• Engineering gotchas (reusable): baukit edit_output hooks MUST be exact 2-arg (output, layer_name) closures — extra default-kwargs are mis-bound positionally by invoke_with_optional_args. The query-position shift at L_read==L_steer is identically zero (cross-position effects only enter via later blocks' attention) — read strictly downstream.

2026-06-11 — FINDING: function is decodable at the query token in FV space, but the label→query map is not low-rank/rotation-like (1-shot)

Phase-1 of the function-geometry experiment (Stream E; scripts capture_oneshot_paired.py + analyze_oneshot_geometry.py; results in results/oneshot_paired_analysis/). Same-output-word, different-function paired 1-shot prompts; source = demo label token, target = query final token.

• Function fingerprint is present and linearly decodable at the QUERY token along the FV difference axis (fv_f1−fv_f2): peak L11 AUC 0.941 / d 2.22 (antonym↔synonym), 0.894 (landmark↔park), mid-layer band L9–15. This is the robust, positive result.
• The label-token difference = one dominant shared axis + a broad high-dim tail. STABLE rank (Σσ²/σ₁²) of the L11 D_label matrix is 5.2 (antonym↔syn, 544×4096) / 3.3 (geography, 84×4096): σ₁ alone holds ~19%/~30% of the energy → a single clear "function axis." BUT 90% of the energy needs k≈315 dims (antonym↔syn) — the residual is high-dimensional/per-word. Stable rank is lowest at mid layers (min ~4.7 @ L9, ~5 through L15), coinciding with peak query-token FV separation; rises to ~9 by L24. (The entropy "effective rank" was ~150; the two disagree precisely because of the fat tail — stable rank keys on the dominant axis.) Centered stable rank (mean axis removed) = 18.8 / 9.4. Robust to magnitude: unit-normalizing each difference vector before stacking leaves stable rank essentially unchanged (5.4 / 3.5 vs raw 5.2 / 3.3) — so the dominant axis is DIRECTIONAL, not driven by a few high-norm vectors. Magnitudes are themselves tight (CV ~23%/18%) BUT 14/544 antonym↔syn rows are exactly zero (degenerate: demo input collided across functions, e.g. borrow←lend in both → identical prompts; excluded when normalizing → 530 rows). Artifacts: D_label.pt, fig_Dlabel_svd{,_unit}_L11.png, fig_Dlabel_magnitude_hist_L11.png, fig_Dlabel_stable_rank_by_layer{,_unit}.png.
• But the demo-label → query-token linear map is weak / high-rank / not rotation-like (held-out R² ≤0.16 mid-layers, eff rank ~190–280, Procrustes gap large). So the original "low-rank manifold + predictable rotation from label to next position" hypothesis is not supported in 1-shot — even though a single dominant function axis DOES exist at the label token.
• Method notes (reusable): (1) hold the OUTPUT token fixed and vary function → the source-token activation difference is pure contextualization (the label token is literally identical across f1/f2; assertion enforced). (2) For a W-sample map into d=4096 with W≪d, the full-matrix M structural metrics (eff-rank, reduced-rank, ‖MᵀM−I‖) are rank-limited/regularization-dominated — trust held-out map_R2 and the FV-separation AUC, not in-sample reduced-rank R². (3) Project the source→target map into the ≤2W-dim data span before any d×d op — all reported metrics are invariant, ~30× faster.
• Decision for Phase 2 (if pursued): steer along the FV difference axis at/near the query token (well-motivated by the separation); do NOT assume steering the demo label token propagates predictably to the query (Phase 1 contradicts it). A multi-shot variant (stronger function identification) is the natural next probe before drawing strong conclusions.

2026-06-12 — Criterion for "confusable task pairs": matched input AND output marginals

For the mixed-task ICL / function-geometry substrate, a good pair (f1, f2) must encode the function ONLY in the (input, output) relation: neither the input token alone nor the output/label token alone may be predictive of which task it is. Formally P(input|f1)=P(input|f2) AND P(output|f1)=P(output|f2). Shared input domain + same output type is NOT enough — the output marginal must match too.

This rejects pairs that looked superficially fine:

• capitalize_first ↔ last: output leak (first- vs last-letter distributions differ).
• english-fr ↔ -de, country-capital ↔ -currency, person-occupation ↔ -sport: output leak (you can ID the task from one output token: a language / a city vs currency / etc.).
• hypernym ↔ hyponym: input leak (specific vs general words) — inverse pairs only satisfy the criterion over a symmetric domain (input set = output set).

Two construction recipes that satisfy it by design: (1) inverse functions over a symmetric/closed domain (next/prev over a cycle; +k/−k; Caesar ±n); (2) two distinct relations whose outputs land in the same word pool as the inputs (antonym/synonym; synonym/rhyme). Operational test: a probe predicting the task from input-token-only or output-token-only embeddings should be at chance.

A second, sharper constraint — in-context learnability — makes the design space much smaller than it looks. Matched output marginals already require the two functions to share an output pool (a lone output token can't reveal the task). That alone kills factual-attribute pairs (capital vs largest-city → disjoint answer sets), translation (disjoint by language), and morphology (suffix leak). On top of that, the function must be inferable from ~5 ICL demos — which kills otherwise-legal ideas like alphabetical neighbor word (model can't know the candidate lexicon). What survives BOTH = "a general rule the model already knows, over a domain with a shared output pool" → arithmetic on numbers, or meaning/sound relations over words. That is exactly the antonym/synonym/ rhyme/number region — the apparent "everything looks similar" is the constraint, not a lack of imagination. Decision: ship 3 pairs (antonym|synonym, synonym|rhyme, antonym|rhyme, next_number|prev_number); do NOT force a 4th. Letters were generated then dropped (26-cap + no new legal+learnable axis). If more variety is ever needed, the only honest routes are to relax a constraint deliberately and measure the cost (e.g. morphology with the input/output leakage probe), or curate an association corpus offline.

Datasets live in dataset_files/paired_tasks/ (generator dataset_files/generate/create_paired_tasks_datasets.py): next/prev_number (words), rhyme (CMUdict, outputs in syn/ant vocab), plus copied synonym/antonym. Note: number-words avoid digit clumping but compound forms >20 are multi-token (word-based) — score at first/last label-token positions.

2026-06-12 — Geography under the matched-marginal rule: only spatial place→place survives

Geography mostly LEAKS: (a) attribute relations (country→capital vs →currency vs →continent) have disjoint output pools → output-token leak; (b) entity-type→country relations leak via the input — verified the repo's park-country has "Park"/"National" in 747/749 inputs, so landmark|park is identifiable from the input alone (why Stream E could only use it geometry-only). The single legal family is place→place spatial relations over one shared pool — the geographic analog of next/prev_number. Built east_neighbor / west_neighbor (dataset_files/generate/create_geography_neighbor_datasets.py): country → nearest country in the E/W bearing quadrant, from most-populous-city coords (geonamescache, offline; deterministic). Both country→country → matched input AND output marginals; identical input set (244 countries with both an E and W neighbour). Multi-token country names allowed (first/last label-token, like compound numbers). Approximate for island/edge countries (e.g. Bermuda→Cabo Verde) — inherent to nearest-in-quadrant. North/south and bordering (needs adjacency data, not available offline) are possible extensions. Installed pycountry-convert (continents) + geonamescache (city coords), and nltk+WordNet (unused so far — semantic co-hyponym/hypernym/meronym pairs remain an option).

General principle (what makes geography/periodic-table work): take a domain the model knows that has internal STRUCTURE (1D order, 2D grid, hierarchy) and pair two "neighbour"-type relations over it; the shared pool + the structure give matched marginals for free. Instances: numbers (1D line: next/prev), geography (2D map: east/west), periodic table (2D grid: next_in_period=right / next_in_group=below). Built next_in_period/next_in_group via mendeleev (offline): 66 main-group+transition elements with both a right and below neighbour, identical input set, multi-token names allowed. f-block excluded (no group_id; model knows them worst). Skipped US-presidents (ordinal over a known list, ~46 — too small). Open structured domains not yet built: north/south geo neighbours, WordNet taxonomy (co-hyponym; is-a vs has-part).

2026-06-10 — Mixed-task ICL probe: synonym pays for diluted demos, not antonym

Probe (src/eval_scripts/mixed_icl_antonym_synonym_topk.py): n=200 prompts, 10 ICL demos = 5 antonym + 5 synonym, query from one task; top-1/2/3 = correct first token in model's top-k. Ran both demo orders + a same-task 10-shot reference overlay. (Query word is excluded from the ICL demos; n=50 SE≈0.06 was too noisy → use n≥200.)

• Antonym query ≫ synonym query (top-1 0.47–0.55 vs 0.12–0.20), every cut. This is task difficulty — synonyms have many valid answers so the single gold first-token frequently misses top-3. First-token top-k understates synonym competence; use any-acceptable-synonym scoring if synonym is the focus.
• Recency is real (~+0.08 top-1): each task scores higher when its demo block is the last one before the query (antonym 0.47→0.55, synonym 0.12→0.20). An earlier n=50 run wrongly called recency "weak/inconsistent" — that was sampling noise.
• Antonym is fairly robust to halved demos (mixed 0.47–0.55 vs same-task ref 0.565, matching the ref when its demos are recent); synonym degrades hard (mixed 0.12–0.20 vs ref 0.365) → synonym pays most for diluted demos.

Repro convention for model-loading scripts on this box: deps not preinstalled — pip install transformers==4.49.0 accelerate (per fv_environment.yml); baukit is not on PyPI, so scripts that only need eval_utils' pure helpers should inline them rather than import the baukit-laden module. GPT-J weights cached under /workspace/.cache/huggingface; run with HF_HOME=… HF_HUB_OFFLINE=1.

---

Conventions (observed in the codebase)

• Model: GPT-J (29 layers, d=4096). FP16 + Qwen variants exist (see git log).
• Task split: task_splits/abstractive_train_test_tasks_29.json = 20 train + 9 test.
• Results layout: one dir per experiment under results/; per-task subdirs hold .pt artifacts + a *_metadata.json / *_summary.json describing the run config.
• Head set artifact: multitask_top_aie_heads.pt (+ _metadata.json) — ranked (layer, head) list by prompt-weighted CIE. Stored top-40; top-10 typically used.
• FV artifact: {task}_function_vector*.pt = sum over selected heads of out_proj applied to the head's mean last-token activation.
• _multitask_top10 suffix on a result/FV dir means it was built from the multitask top-10 head set rather than per-task FVs.
• Save intermediates (within storage reason). When a big experiment has an expensive stage (forward passes, CIE, activation capture), persist the general intermediate — not just the final answer — so variations rerun in minutes instead of hours. Concretely: store the full per-(layer,head) CIE grid + a generously-sized ranking (top-40, even if top-10 is used) rather than only the chosen heads; store mean activations for all heads/positions needed by any plausible N/k, not just the selected ones; keep per-prompt effects when subset re-aggregation might matter; cache residual-stream activations once and reuse across regression targets. Payoff observed 2026-06-11: top-20/30/40 varicl FVs + a whole new PCA-ridge heatmap cost minutes because stage-1 saved all-head activations and the top-40 ranking. Guardrail: this is for O(GB) per-task tensors and grids, not for dumping every forward pass — if an intermediate would cost ≫ recomputing it on demand, skip it and note the recompute command in the metadata instead.

---

Decided

2026-06-12 — FVs round 2: east/west_neighbor + next_in_period/group — built, but LOW-N

Same 3-method derivation as the trio below (manifest now task_splits/paired_tasks_7.json; builder manifests fv_manifest_paired2.json; auto no-filter retry available but unused — all 4 tasks pass the filter). All 12 FVs in the organized folder, norms 24–44. CAVEAT: GPT-J is weak at all four — ICL-correct counts: canonical 10-shot east 5/51, west 7/51, period 2/14, group 5/14; varicl east 2, west 3, period 1, group 1 → the correct-only mean activations (esp. varicl, 1–3 prompts) are HIGH-VARIANCE. Filter kept ON for consistency with the 29 originals. If these tasks matter downstream, prefer no-filter variants or bigger datasets. (Effective-n hierarchy among paired tasks: next/prev_number excellent (42/42) ≫ elements/geo (1–7) ≥ rhyme (0 → no-filter).)

2026-06-12 — FVs derived for the 3 paired tasks (rhyme, next_number, prev_number) × 3 methods

The new dataset_files/paired_tasks/ trio now has FVs under all of task_specific, train_selected, train_varicl (organized folder + underlying caches; fv_manifest_paired.json in each builder root so the original 29-task manifests are untouched). Conventions adopted:

• The 3 task JSONs are symlinked into dataset_files/abstractive/ (user-approved; the loader hardcodes abstractive/extractive). Split seed 42 → 140/18/42; valid is only 18.
• task_splits/paired_tasks_3.json = manifest for stage-2 builders (they validate --tasks).
• rhyme FVs are NO-FILTER (--no_filter_to_correct_icl, both regimes): GPT-J is 0/18 ICL-correct on rhyme valid, so the standard correct-only averaging is impossible. Treat rhyme's FV as "attempted-task" activations; flag in any cross-task comparison. next_number (18/18) and prev_number (15/18) are standard.
• Repro trick: running varicl stage-1 for extra tasks with --num_shards N>1 keeps writes_global=False → per-task files only, existing pooled head artifact safe. We still used an isolated root (results/multitask_aie_heads_varicl_paired/, kept for provenance) and copied the *_mean_head_activations_varicl.pt into results/multitask_aie_heads_varicl/<task>/.

2026-06-11 — PCA-space (direct) activation→FV ridge sweep + 16-PC vs full-dim comparison

Companion to the full-dim ridge decoder (Stream C below), run in a 16-PC bottleneck across all 31 token positions × 29 layers. Per cell: activation PCA (k_act=16) fit per-cell on 20 train; FV PCA (k_fv=16) fit once on 20 train FVs; ridge 16→16, λ by leave-one-train-task-out CV, single 20-train standardizer; predict 7 test tasks, reconstruct to 4096-d, score there. Direct projection (not the deprecated joint). FV target train_selected. cc/pc excluded.

• Scripts: NEW regress_activation_to_fv_pca_ridge.py; launcher run_fulldim_ridge_shards.sh generalized with a WORKER env override (drives both sweeps); merge reused unchanged.
• Output: results/pca_ridge_activation_to_fv/ (combined_metrics.csv = 899 rows + heatmaps).
• HEADLINE: the 16-PC bottleneck is free at the optimum — PCA best icl10/finaltok @ L13 = 0.1147 vs full-dim @ L11 = 0.1161 (PCA marginally better; it denoises). In the mid-layer sweet spot (L8–13) PCA ≈/< full-dim; in later/embedding layers PCA is worse (L28 0.158 vs 0.149). Net mean Δ(pca−full)=+0.003. → 16 activation PCs hold all recoverable activation→FV signal where it's concentrated; the regression target genuinely lives in a ~16-dim subspace.
• Metric identity: fv_test_mse = (k_fv/4096)·pca_test_mse + floor (FV-PCs orthonormal), verified to 4e-8. FV-PC reconstruction floor (test) ≈ 0.099. So selecting α on PCA-space CV MSE == selecting on reconstructed MSE (differ by the constant floor).
• Comparable metric note: this reconstructed-4096-d MSE is the same unit as the full-dim ridge (0.116) and the k/layer sweeps — NOT the joint-PCA-space MSE (Open Q3).

2026-06-10 — train_varicl RAN to completion; GOTCHA: stage-2 needs test-task activations precomputed

The variable-ICL method is fully built for GPT-J: 29 FVs + heads.pt + manifest + per-task selected_heads.json under results/function_vectors/gpt-j/train_varicl/. Pooled top-10 (varicl) = [(9,14),(15,5),(8,1),(12,10),(11,0),(8,0),(14,0),(24,6),(21,2),(10,0)]; 8/10 overlap with train_selected (top-2 identical), so the variable-ICL regime selects nearly the same head subspace as fixed-ICL train pooling — overlap, not identity, as expected.

GOTCHA (pipeline gap, now handled): the documented step sequence (stage-1 --task_split_key train_tasks → stage-2 compute_all_task_fvs_varicl.py) is INCOMPLETE. Stage-2 builds FVs for all 29 tasks and requires <task>_mean_head_activations_varicl.pt to already exist for every task (it raises FileNotFoundError, does NOT compute on the fly). Stage-1 over train_tasks only writes the 20 train tasks' activations, so the build crashes on the first test task. Fix: run stage-1 once more on the test split to an isolated dir (so writes_global can't clobber the train-pooled head artifact), then copy the 9 activation files in:

python src/eval_scripts/compute_multitask_varicl_heads.py --task_split_key test_tasks \
  --abstractive_only --query_split valid --demo_split train --n_top_heads 40 --batch_size 8 \
  --min_shots 1 --max_shots 10 --max_successful_prompts 170 --filter_to_correct_icl \
  --save_per_prompt_effects --save_path_root results/_varicl_testtasks --num_shards 1
# then: cp results/_varicl_testtasks/<task>/<task>_mean_head_activations_varicl.pt
#            results/multitask_aie_heads_varicl/<task>/   (for the 9 test tasks)
python src/eval_scripts/compute_all_task_fvs_varicl.py --overwrite

This recomputes CIE for the 9 test tasks too (unavoidable; they're short-sequence so it's cheap and the resulting test-pooled head artifact is harmless/unused). Future fix option: teach stage-2 to compute missing activations itself, or have the runner also process test_tasks for activations only.

2026-06-10 — 4th FV method: train_varicl (variable-ICL, train-pooled) — IMPLEMENTED, not yet run

A new head-selection + FV method alongside task_specific / train / train+test. Same two-stage shape, but the prompt regime changes:

• Variable ICL: each prompt draws a random 1–10 demonstration count (deterministic per (task_index, query_idx); shard-invariant via global task_index).
• Correctness filter, capped: keep only correctly-answered prompts, ≤170 per task.
• Read position: mean head activations AND the CIE intervention both read the query predictive (last) token, T=-1 (user-confirmed). Single, length-independent position → mean activations stored as (n_layers, n_heads, head_dim) (no token axis), so a new save filename <task>_mean_head_activations_varicl.pt and adapted indexing are required (the canonical builder's [...,-1] would grab a scalar — see the three corrections below).
• CIE: variable ICL + shuffled labels, intervention at the query token, on the same correctly-answered query set; CIE prompts use seed args.seed + cie_seed_offset (default 500000) so they differ from the activation pass but stay reproducible.
• Pooling: CIE averaged across the 20 train tasks (same as train_selected), top-N heads.

New sibling scripts (existing engines untouched): src/utils/varicl_utils.py, src/eval_scripts/compute_multitask_varicl_heads.py, src/eval_scripts/compute_all_task_fvs_varicl.py, src/eval_scripts/run_multitask_varicl_all_tasks.sh. New args: --min_shots, --max_shots, --max_successful_prompts, --cie_seed_offset.

Three corrections vs the canonical code (handled): (1) the fixed-n_shots correctness filter can't vary shots → new varicl_correctness_filter; (2) CIE avg_activations[L,H,token_idx] → avg_activations[L,H] for single-position; (3) FV builder mean_activations[L,H,-1] → mean_activations[L,H].

Outputs (when run): results/multitask_aie_heads_varicl/ (head set + per-task CIE/activations); results/function_vectors/gpt-j/train_varicl/ (FVs + heads.pt + selected_heads.json). Run commands

• smoke verification: WORKLOG Stream D and plan /root/.claude/plans/immutable-finding-boole.md.

2026-06-10 — Direct full-dim (4096→4096) activation→FV ridge decoder (Stream C)

New experiment line, deliberately PCA-free (distinct from regress_activation_to_fv_joint_pca*). For each (token position, layer) cell, fit one ridge map from the raw 4096-d residual activation to the raw 4096-d train_selected FV. λ chosen by leave-one-train-task-out CV (20 folds); a single standardizer fit on the pooled 20-train rows is reused everywhere; MSE reported natively in 4096-d on the 7 test tasks (cc/pc excluded). 31 token positions × 29 layers = 899 cells.

• Scripts: regress_activation_to_fv_fulldim_ridge.py (worker, one shard = one ICL idx; GPU ridge via eigendecomposition reuse so the α grid is ~free), run_fulldim_ridge_shards.sh (tmux sharding, round-robin), merge_fulldim_ridge_results.py (combined CSV + heatmaps + summary).
• Outputs: results/fulldim_ridge_activation_to_fv/.
• GOTCHA (now handled): in the 4tokens dir the final prompt token (last_prompt_token) has icl_example_index = None; the 3 label roles use 10. Filter the final-token role on None.
• Result: best decode = final prompt token @ layer 11, test_mse 0.116; clean layer bowl min at L11 (band L10–14); query position (ICL 10) beats earlier ICL demos. Embedding-layer pre-label/final tokens are constant across tasks (→ predict-the-mean baseline 0.217).
• Metric note: this 4096-d MSE is NOT comparable to the joint-PCA-space MSE (Open Q3); it is comparable to the reconstructed-4096-d MSE used by the k/layer sweeps.

2026-06-10 — Three head-selection methods + organized FV folder

FV derivation is two stages: (1) select top-N heads by CIE; (2) per task, sum out_proj(mean_head_activation[L,H,-1]) over the selected heads. The three methods differ only in stage 1 — which tasks' per-task CIE is pooled to rank heads. Stage 2 (mean head activations) is shared. All three use top-10, GPT-J-6B, query_split=valid, demo_split=train → directly comparable.

Method	Stage-1 pool	Head set	Per-task FVs
task-specific	1 task	gptj_fv/<task>/<task>_indirect_effect.pt	gptj_fv/<task>/<task>_function_vector.pt
train	20 train tasks	multitask_aie_heads/multitask_top_aie_heads.pt	gptj_fv_multitask_top10/ (29)
train+test	29 tasks	multitask_aie_heads_all_tasks/multitask_top_aie_heads.pt	BUILT → function_vectors/gpt-j/train_test_selected/ (29 + manifest)

Organized access point: results/function_vectors/{task_specific,train_selected,train_test_selected}/ — a clean view with uniform <task>_function_vector.pt naming so any --fv_root points straight at a method. See results/function_vectors/README.md.

Symlinks, not hard moves. results/gptj_fv is dual-role (task-specific FVs + the shared *_mean_head_activations.pt cache + CIE + eval JSONs) and ~15 scripts default to the old paths. Hard-moving would break the pipeline, so the FV folder symlinks into the existing caches. Underlying dirs remain source of truth. Reversible; revisit if we want a true single-source-of-truth migration (would require rewiring all --fv_root defaults).

2026-06-10 — k-sweep outputs nested under results/k_sweeps/

All activation→FV k-sweep result dirs now live under results/k_sweeps/ (was flat in results/). Names trimmed of the redundant k_sweep_ prefix (the parent conveys it):

• results/k_sweeps/activation_to_fv_ols_multitask_top10_log2/ — original, 9 test tasks.
• results/k_sweeps/activation_to_fv_ols_multitask_top10_log2_exclude_cc_pc/ — 7 test tasks (drops country-currency, product-company). Default --output_dir in sweep_k_activation_to_fv_ols_log2.py and sweep_k_activation_to_fv_ols.py updated to match; run_config.json self-paths fixed.

2026-06-10 — Direct k-sweep, third axis: fix k_act, sweep k_FV → joint optimum (16,16)

sweep_k_activation_to_fv_direct_log2.py now has --fix_act_k K (mutually exclusive with --fix_fv_k): pins k_activations=K and sweeps k_FV (doubling grid reinterpreted as k_FV, capped at fv_k_cap). Run at k_act=16 (7 tasks): test MSE falls monotonically with k_FV, best at the k_FV=16 cap (14/15 series), riding just above the recon floor (gap ~0.01–0.03). No k_FV overfitting — k_FV is bounded by discarded-FV-variance (floor) and by the FV-PCA rank cap (16), not by overfitting. With 16 activation PCs the regression recovers any k_FV target nearly to the floor. Combined over all three direct cuts (coupled diagonal / fix k_FV / fix k_act): the joint optimum is (k_act≈16, k_FV=16) — the corner. k_act peaks ~16 then overfits; k_FV climbs monotonically to its 16 cap. The four direct dirs in results/k_sweeps/: ..._log2, ..._log2_exclude_cc_pc, ..._log2_fixedfvk16_exclude_cc_pc, ..._log2_fixedactk16_exclude_cc_pc.

2026-06-10 — All k_sweeps are DIRECT; joint results removed

Per user decision, the k-sweep result dirs are now direct-method only. The three joint runs were deleted and replaced by direct equivalents; results/k_sweeps/ holds exactly:

• activation_to_fv_direct_ols_multitask_top10_log2/ — coupled diagonal, full 9 test tasks.
• activation_to_fv_direct_ols_multitask_top10_log2_exclude_cc_pc/ — coupled diagonal, 7 tasks.
• activation_to_fv_direct_ols_multitask_top10_log2_fixedfvk16_exclude_cc_pc/ — k_FV pinned 16, sweep k_act, 7 tasks. The joint script (sweep_k_activation_to_fv_ols_log2.py) is retained as a tool; only its results were removed. Settling finding (direct, k_FV=16): k_act=1 → test MSE ≈ 0.18–0.19 (≈ predict-the-mean ~0.21), k_act=16 → ≈ 0.126. A single activation-PC does NOT recover the 16-dim FV — the joint method's apparent "1 PC suffices" was the 16 appended FV-basis features, not genuine low-rank recoverability. Optimal remains k_act≈16 (first-label 32), k_FV=16, ICL5.

2026-06-10 — Regression definition: "direct" k_activations→k_FV (vs "joint")

Two regression definitions now coexist for the activation→FV PCA decode:

• joint (sweep_k_activation_to_fv_ols_log2.py + the regress_*_joint_pca* family): X and Y both projected onto the SAME concatenated [act-PCA (k_act) | FV-PCA (k_fv)] basis; OLS maps the (k_act+k_fv)-dim joint projection of the activation → the joint projection of the FV. The FV-PC half of the prediction is reconstructed for the 4096-d MSE.
• direct (NEW sweep_k_activation_to_fv_direct_log2.py): X = activation→act-PCA (k_act) only; Y = FV→FV-PCA (k_FV); OLS regresses R^{k_act}→R^{k_FV}; predicted FV-PCs reconstruct to 4096-d.

Why they differ: joint's feature vector additionally includes the activation projected onto the FV basis (k_fv extra inputs), so joint fits slightly better. Measured: direct is ~0.002–0.003 higher test MSE than joint at the optimum (icl5/first 0.1261 vs 0.1232). The direct setup is the cleaner "regress activation space → FV space" and should be preferred as the headline definition; joint numbers were mildly optimistic. Optimal k (≈16, first-label 32), ICL monotonicity, and token-role ordering are all unchanged between the two. The new script mirrors all log2 flags incl. --fix_fv_k.

2026-06-10 — k is two knobs: k_activations vs k_FV (--fix_fv_k)

The joint-PCA regression has two independent dimensionalities: k_FV (FV-target PCs) and k_activations (input PCs). The original log2 sweep coupled them below the cap (fv_k = min(k, 16)), conflating the two. New flag --fix_fv_k on sweep_k_activation_to_fv_ols_log2.py holds k_FV at fv_k_cap for every k, isolating k_activations. Finding (k_FV=16): first/last-label tokens optimal at k_activations≈16–32; pre-label token optimal at k_activations≈1–8 (more PCs hurt — little task signal there). Beyond ~16–32 the activation side overfits regardless of token. Pinning k_FV=16 only changes the k<16 regime vs the coupled run (it's strictly better there since the FV target isn't shrunk). Canonical config: layer 11, last/first-label, ICL 5, k_activations≈16, k_FV=16.

2026-06-10 — Excluding cc/pc barely changes the regression k-sweep (FINDING)

Re-ran the log2 sweep without country-currency + product-company (whose train-multitask-selected FVs steer much worse than task-specific). The activation→FV regression test-MSE moved by only ~±0.001 and the optimal-k structure (k≈16–32, fv_k 16, overfit past 32) was unchanged. Implication: "bad for steering" ≠ "hard to regress from activations" — the two task-quality notions are decoupled, so the cc/pc weakness is not visible in (and not fixable via) this reconstruction-MSE metric. If their weakness matters, measure it on the steering/FV-effectiveness side (cf. evaluate_heldout_multitask_head_fvs.py).

2026-06-10 — Split provenance of all-tasks head set: RESOLVED (was Open Q1)

The all-tasks head set was computed on query_split=valid (not train). Evidence: per-task files are suffixed _valid (*_mean_indirect_effect_over_valid.pt, *_per_prompt_indirect_effect_valid.pt) — the suffix is emitted from the actual split — and the runner passes --query_split valid. The metadata top-level query_split=train is a stale default, almost certainly written by the --reduce step (which didn't receive the arg). Both head sets therefore share query=valid, demo=train and are comparable. Follow-up (low priority): fix the --reduce path in compute_multitask_top_aie_heads.py to record the true split.

DONE — train+test function vectors built (GPT-J)

Built 2026-06-10 with (reuses cached activations; zero forward passes):

python src/eval_scripts/compute_all_task_fvs_from_multitask_heads.py \
  --heads_path results/multitask_aie_heads_all_tasks/multitask_top_aie_heads.pt \
  --n_top_heads 10 --fv_root results/gptj_fv \
  --output_root results/function_vectors/gpt-j/train_test_selected \
  --task_manifest task_splits/abstractive_train_test_tasks_29.json

→ results/function_vectors/gpt-j/train_test_selected/<task>/<task>_function_vector.pt (29) + fv_manifest.json. All three methods now complete for GPT-J. The builder also supports --tasks (subset shard) and --manifest_name for future models with no cached activations (each shard runs prompts at batch size).

2026-06-10 — Per-task head metadata (selected_heads.json)

Every task folder under results/function_vectors/<model>/<method>/<task>/ carries a selected_heads.json listing the [layer, head, mean_indirect_effect] heads that built its FV, plus selection_pool. Generated (idempotent) by:

python src/eval_scripts/write_fv_head_metadata.py --model_root results/function_vectors/gpt-j --n_top_heads 10

task_specific reads each task's own top_heads (unique per task); the two multitask methods read the shared heads.pt (top-N). Rerun after building FVs for a new model.

2026-06-10 — IMPORTANT: train vs train+test is degenerate at n=10

At top-10, train_selected and train_test_selected select the same head set (same membership; rank order and CIE scores differ slightly). Since the FV is an order-independent sum over selected heads, the function_vector tensors are exactly equal for all 29 tasks (verified: global max|Δ|=0; norms match). NB the .pt files are not byte-identical — cmp differs on dict metadata (top_heads order, paths) — only the tensors match. They diverge only at larger n (34/40 overlap at n=40; first differences ~n=11+). Implication for experiments: a train-vs-train+test comparison at n=10 measures nothing — raise n_top_heads (rebuild with a larger --n_top_heads) or use a per-task metric to study the selection-leakage effect. task_specific is genuinely distinct from both at every n.

---

Open questions (coordinator-flagged 2026-06-10)

1. Canonical head set for held-out eval. evaluate_heldout_multitask_head_fvs.py used the train-only (20-task) head set. Should held-out eval use the all-tasks set instead, or both as a comparison?

2. One FV builder, not three. compute_task_fv_from_multitask_heads.py, compute_all_task_fvs_from_multitask_heads.py, and compute_fv_from_selected_heads.py overlap. Leaning: compute_all_task_fvs_from_multitask_heads.py is the canonical batch builder (one model load, all 29 tasks, standard <task>_function_vector.pt filenames + fv_manifest.json) — it's the one in the build command above. compute_fv_from_selected_heads.py is the flexible single/ad-hoc builder (arbitrary head source, --tasks, non-standard _fv_<tag>.pt filename). Mark compute_task_fv_from_multitask_heads.py (single-task) legacy.

3. Canonical regression metric. Joint-PCA-space MSE vs reconstructed 4096-d FV-space MSE are not comparable. Pick one as the headline metric; report the other as secondary only.

4. Canonical FV target for regression. Some regression scripts target gptj_fv, others gptj_fv_multitask_top10. Decide which is the primary target (now also reachable as function_vectors/task_specific vs function_vectors/train_selected).

5. Standardize ICL index range across regression/sweep scripts (currently mixes 1–4 and 2–5).

6. Save per-prompt CIE effects on the baseline too? The train-only head set lacks per-prompt effects, so select_heads_from_cie_subset.py cannot re-subset it. If subset analysis matters for the baseline, re-run with --save_per_prompt_effects.

7. Smoke-test dirs. results/joint_pca_activation_to_fv_regression_smoke and results/pca_abstractive_fv_activation_scatter_smoke — keep or remove?

---

Process notes

• Workers: register a stream in WORKLOG.md before editing; don't co-edit the same source file without coordinating (per CLAUDE.md). compute_multitask_top_aie_heads.py and evaluate_heldout_multitask_head_fvs.py are the currently-modified shared files.
• Nothing is committed yet; all experimental scripts + results are uncommitted.

2026-07-13 — Verify token positions against the tokenizer, not by offset arithmetic

The 10-shot strip studies selected the "input" slot as pre_label − 1, silently landing on the constant "A" template token for months of grids (labels carry their leading space, so the input word is 2+ tokens further left). Lesson: when a script picks "the X token" by positional arithmetic, print/assert the decoded token string for a sample prompt at startup (or derive positions from the get_token_meta_labels roles, e.g. demonstration_{i}_token), never trust offsets from adjacent structural tokens. Fixed in both steer_tenshot_strip_{cos,norm}_heatmap.py (see WORKLOG Stream R).

2026-07-14 — Two-shot token-pair study: perpair steering mode + the d_in bug applied here too

• steer_twoshot_tokenpair_cos_heatmap.py --steer_mode perpair = per-pair (matched-counterfactual, the user's "nearest neighbour") steering: each source prompt is injected with its OWN pair's tgt−src activation diff at the edited (token, layer); α=1 ≡ exact single-site activation patching (asserted at runtime). Results root: twoshot_tokenpair_perpair_cos_heatmap/ (α∈{0.5,1,2}).
• The 2026-07-13 "verify token positions" lesson applied to this script as well (contrary to Stream R's "two-shot study unaffected" note): its input2/qinput were pre_label−1/qfinal−1 = the constant "A". Consequently in the ORIGINAL twoshot_tokenpair_intervention_cos_heatmap/ results, every grid whose source OR read token is input2/qinput measured the "A" token — do not cite those; grids among {label1, prelabel2, label2, qfinal} are unaffected. A position-correct mean rerun (α∈{2,4}) lives in twoshot_tokenpair_mean_fixedpos_cos_heatmap/; the original dir is kept until the user decides to replace it.

2026-07-14 — Cumulative clamp mode for the two-shot token-pair study

steer_twoshot_tokenpair_cos_heatmap.py --layer_mode cumulative (perpair only) hard-clamps the intervention token to the matched counterfactual's activations at every layer from the start layer i through 28 — per-pair trajectory patching, no strength sweep (α irrelevant under clamping). File/figure names carry a NOMINAL alpha1 tag purely for plot-script compatibility (plot_twoshot_tokenpair_heatmap_grid.py --alphas 1); do not read it as a strength. The grid x-axis is the clamp START layer, and the clean-token embedding-column≡0 invariant does NOT apply in this mode. Results: twoshot_tokenpair_perpair_cumclamp_cos_heatmap/.

2026-07-14 — METRIC OF RECORD for all steering "cos" studies: dircos (Δcos-to-target DEPRECATED)

Per user (emphatic): the intended steering-impact metric was always the DIRECTION-ALIGNMENT cosine dircos = mean_pairs[ cos( act_tgt − act_src , act_src_steered − act_src ) ] (at the read site) — "does the displacement caused by the intervention point along the counterfactual direction" — NOT the previously-implemented Δcos-to-target (cos(steered,tgt) − cos(src,tgt)), which is bounded by 1 − baseline (~0.1–0.2 on near-identical paired prompts) and conflates alignment with proximity.

• Every cos grid/figure/summary number computed BEFORE 2026-07-14 is in the deprecated metric — do not cite (this includes Stream Q/R's tenshot "input rows ≈ 0" cos numbers and all prior twoshot token-pair results). All grids under twoshot_tokenpair_*_cos_heatmap/ and tenshot_strip_intervention_cos_heatmap/ were recomputed as dircos on 2026-07-14 (summary JSONs carry a "metric" key; old data recoverable from git history).
• Convention: structurally-unaffected cells (zero displacement, e.g. read layer ≤ edit layer) render as 0 (F.cosine_similarity's eps does this naturally); dircos ∈ [-1,1] ⇒ scalar-overview panels use symmetric diverging scales, not one-sided Reds.
• Lesson: confirm the METRIC DEFINITION with the user before building/extending measurement studies — the formula in a docstring is not evidence the user wants that formula.

2026-07-16 — Propagated ablation as a MODE of the perlayer script; pod CUDA filter; same-GPU regression tests

• The propagated fixed-direction ablation (Stream X3) is --mode propagated on ablate_oneshot_preimage_logprob.py, NOT a separate script: only the direction rule (fixed U[L] vs per-layer U[b]), the edited positions (anchor + all later tokens vs anchor only), and the default output root differ; prompts/arms/cf-map/sanity checks are shared. Modes write to SEPARATE results roots (oneshot_preimage_ablation vs ..._propagated) so the summary rebuild never mixes them. Follow this pattern for future edit-rule variants.
• RunPod: always pass allowedCudaVersions: ["12.8","12.9","13.0"] in podFindAndDeployOnDemand for the function-vectors-dev template. The image needs CUDA ≥ 12.8; without the filter the scheduler happily lands on 12.4-driver hosts and the pod sits at uptime 0 / ports null forever (two 4090 pods lost to this on 2026-07-16).
• Regression-test model outputs on the SAME GPU. fp16 GPT-J forwards differ by up to ~1.7e-2 in delta-log-p across GPU architectures (L4 vs prior pod, identical code). A code no-op check must compare old-code-vs-new-code on one GPU (expect exactly 0.0), never new-code-on-GPU-A vs stored-run-from-GPU-B with a tight tolerance.

2026-09-06 — Style-property STEERING is a sandbox: no default result (user decision)

• All steering work under results/style_properties/steering/ is exploratory. There is NO default, headline, or recommended steering technique. Promotion of any variant to repo standard requires an explicit user decision recorded here. Do not describe any cell as "the" steering result, and do not create root-level headline figures in that folder.
• Variant grid (the only organising structure): {meandiff, meanact, sparsehead} × {kall, k4} × {succno, succyes} = 12 cells, named <technique>__<kfilter>__<successfilter>. Registry: src/sandbox/ext_styleprops/variants.py (single source of truth; adding a technique = adding one entry, not a script).
• Fixed shared protocol so cells are comparable: first cue token per document · 200 docs · 32-token T=1 seeded rollouts · LLM coherence judge drops gibberish · primary metric = strict rate with an unscorable rollout counted as NOT adopting · counterfactual + reverse controls expected for every variant · shared sweep grid layers 2-24 × doses 0.5-32. A cell that searched less records it in its spec and is flagged in the comparison figure.
• Comparison presentation: alphabetical, unranked, no best-cell highlight (so the table cannot imply a default).
• Excluded from the grid (kept, not deleted): the pre-2026-09-02 runs at all cue sites with unjudged short rollouts (artifacts/.../_archive_allk), and arms whose vector was derived from EVIDENCE tokens rather than cue tokens — derivation site is not a grid axis.
• LESSON — arm keys are not stable literals: the runner named a property's arm <vec>_cue_nat2alt when that vector won the property's sweep and <vec>_cue_nat2alt_best when it did not, with per-property _L<l>_a<a> shortlist suffixes; and rawalt_cue_cue_* (cue-derived) vs rawalt_cue_* (evidence-derived) differ by one segment. Match arms by REGEX with explicit vector gating, and verify with a losslessness check against previously reported numbers (build_variant_results.py --check) — a literal-key assignment silently mis-assigned three properties before that check caught it.

2026-09-06 — Translation-framed style continuation is a SANDBOX variant (user decisions)

• What it is: an alternative way to teach a style convention in context — the document is given in Spanish (Spanish: header), then an English: section translates it back in the target convention, truncated after the cue token; readout = the plain prescreen's sampled continuation, scored for style AND translation correctness. Lives in results/style_properties/translation_framing/; exploratory, not a headline result.
• Spanish is neutral and fixed (user decision): one Gemini translation per base doc in standard Spanish typography (single spaces, straight quotes, digits, %, sentence caps, parenthetical dashes → -), shared by both English polarities, enforced by a deterministic fix-up and audited (es_audit.csv). The English prefix is the only carrier of the convention. Do not build "mirrored-convention Spanish" variants without a new user decision.
• Translation metric (user decision): Gemini judge PRIMARY (conventions explicitly ignored, gold reference = the document's own next tokens), normalised exact match SECONDARY. Accuracy of the framing = style adopted AND judge-correct, over style-scorable judged items. English-only comparison carries exact match only (free continuation has no translation target) — state the asymmetry wherever the two are plotted together.
• Prompt layout (user decision): language labels Spanish: / English: (not bare text).
• Scope (user decision): only the translation framing is sampled; the English-only curve is the 2026-09-01/02 prescreen record set (same items, seeds differ by the framing tag).
• Mechanics to keep: the header is tokenised separately and PREPENDED to the unchanged English ids so the stored cue index/id assertions still hold; prescreen_adherence.py --add_refs backfills ref_nat/ref_alt/ctx_tail into existing records in build_items order (asserted) rather than re-sampling.

2026-09-07 — Translation framing covers all 17 style properties; pool re-admission is a pending user decision

• Scope extension (user request): the 4 properties excluded by the English-only prescreen (whilst, ellipsis, brit_t_past, ise_ize) were run through the translation framing and the Stage-A4 gate re-evaluated under both framings for all 17 (results/style_properties/translation_framing/gate.csv). The pruning reason (scorable rate below 15 %) does not hold under the framing (0.51–0.69 scorable), because the source anchors content and the model reaches the lexical slot.
• Pool status is UNCHANGED until the user decides. task_splits/style_properties_pool.json still lists 13; ise_ize passes the gate under the framing (n = 118), whilst/ellipsis pass on thin cells (18 / 8), brit_t_past fails on separation. Re-admitting any of them — and whether that applies to the English-only pool too — requires an explicit user decision.
• Gate reimplementation rule: plot_translation_framing.gate_stats must reproduce behavioral_prescreen/prescreen_summary.csv on the English-only records before any new-framing gate number is reported (it does; the nan-at-k=0 prior-pole fallback and constant-curve Spearman = undefined are the two details that had to match plot_prescreen.py).
• Repo hygiene (user-approved 2026-09-07): all worktree-*/claude-* side branches deleted on origin and locally, all worktrees removed; the repo has only main. Background sessions must never call EnterWorktree here (.claude/settings.json has bgIsolation: none); if the Edit-tool guard still refuses in-place edits, write via shell/python instead of creating a worktree. Files another agent has uncommitted in the shared checkout are never staged.

2026-09-07 — Translation-framing study RETRACTED and deleted (user decision); restart from scratch

• All translation-framing data and results are deleted at the user's request: the Spanish corpora (base_corpus_es.json, the k>=5 supplement base_corpus_k5*.json and props_k5/), the sampled records (artifacts/style_properties/prescreen_translate*, prescreen_k5), the results folder results/style_properties/translation_framing/, and the three study scripts (translate_corpus.py, judge_translation.py, plot_translation_framing.py); prescreen_adherence.py is back to its 2026-09-06 form and the backfilled ref_nat/ref_alt/ctx_tail/trans_exact keys were stripped from artifacts/style_properties/ prescreen/*.json. The 2026-09-06/07 DECISIONS entries and WORKLOG findings about that study are void. Git history keeps the code if a redesign wants to reuse pieces.
• Why: the Spanish source was not held to ONE fixed convention — the translation prompt preserved each number's source form, so ordinals were mixed digit/word within and across documents (ordinal_words docs: 177 of 323 mixed). USER PRINCIPLE (must hold in any redesign): the Spanish text is identical for both polarities AND internally in one fixed convention for every feature; the English translation in context is the ONLY thing that varies and the only carrier of the convention. Also: say "English translation so far", never "English prefix".
• Kept (predates the study, other work depends on it): the 878-doc base corpus, the 17 property datasets and registry, the A4 prescreen records, decodability, and the steering sandbox. The pool is still 13.
• Redesign: to be agreed with the user BEFORE any generation or compute.

2026-09-07 — Style-properties line RESET (user decision)

• Everything downstream of the style registry was removed: dataset_files/style_properties/ (base corpus + 17 datasets), task_splits/style_properties_pool.json, all src/sandbox/ext_styleprops/ scripts except properties.py (the 17-style registry, KEPT), and artifacts/style_properties/ (26 GB, gitignored — gone for good). results/style_properties/ was archived with history to archive/style_properties/ (prescreen, decodability, steering sandbox, adjudication memo). STYLE_PROPERTIES_DIR in src/utils/paths.py is kept for the restart.
• Reason: accumulated confounds (translation source not in one fixed convention, resampled number items, thin k-cells, wording confusion) — the user prefers a clean restart over stacked fixes.
• Redesign of the free-form style study is to be agreed with the user BEFORE any generation or compute. Rules carried forward: one fixed convention per feature in any source text (audited), >=100 samples per k for k=0..5 per property, features must not occur inside the first 12 tokens of a document.

2026-09-07 — Style-translation study, step 1: Spanish source corpus conventions and verification rule (user decisions)

• Corpus: dataset_files/style_translation/final/<family>.json = 200 unique verified Spanish paragraphs per family (17 families from src/sandbox/ext_styleprops/properties.py), each with >=5 (aim 5-7) places where the English translation must choose between the family's two styles. The Spanish is identical whatever style is tested later and never carries the tested style.
• Fixed Spanish convention (user): RAE standard with ALL numbers as digits — cardinals 3 días, ordinals in ONE form 3.º/2.ª (apocope 3.er normalised), percentages 15 %, quotes « », rayas attached —inciso—, ellipsis ..., single space, normal case, no markup. Enforced by gen_spanish.fixup, audited by gen_spanish.violations (0 violations in 5,103 candidates). Any future source text for this study must pass the same audit before use.
• Verification rule: LLM judging is ONE text per call with a strict JSON schema (Claude Haiku 4.5 via OpenRouter, T=0): coherent, fluent, anchors -> k_found. Length and conventions are judged deterministically, never by the model. pass = coherent & fluent & k_found>=5 & regex_k>=5 & 0 violations & 100-230 words, then de-duplication. Agent-tool sub-agents handed hundreds of texts wrote heuristics instead of reading (WORKLOG 2026-09-07) — do not use them for bulk text judging.
• Scripts: src/sandbox/style_translation/{families,gen_spanish,verify_spanish,collect_verdicts,calibration}.py; path constant STYLE_TRANSLATION_DATA in src/utils/paths.py. Step 2 (English side / prompts / GPT-J) awaits the user.

2026-09-07 — Style-translation step 2: English house style and twin construction (user decisions)

• Minimal pairs, not two translations (user): one Gemini translation per Spanish text, in a fixed English house style = the nat pole of all 17 families (standard American English; list in dataset_files/style_translation/README.md), enforced deterministically with the registry renderers; the alt twin flips exactly one family via render(..., "alt"). Twins are identical outside the opportunity spans (checked, 0 diffs).
• Judge scope (user): Haiku checks only the initial translation; gates = coherent + faithful. Style/conventions are judged deterministically (k_en >= 5, audit_nat == []) — the judge hallucinated style verdicts, as it did for the Spanish conventions. fluent and style_consistent are recorded, not gating.
• Registry fixes (properties.py): Contractions skips the modal "have to"; NumWords/OrdinalWords capitalise a spelled-out word that opens a sentence. Any later scoring uses the same detectors, so this is the definition from now on.
• Corpus of record for step 3: dataset_files/style_translation/pairs/<family>.json (200 per family; opps[k].nat_span/alt_span are the cue sites). english/ keeps all attempts; final/ stays the Spanish-only step-1 output.

2026-09-07 — Paper figure house style (user decision: "use this type of style" for all paper visuals)

• Schematics (circuit, method, decomposition explainers) are TikZ vector figures in write_up/graphics/, sharing figstyle.tex (Inter text, CM math, palette, \vstack/\bstack residual-cell macros). Build: pdflatex <name>.tex && pdftoppm -r 300 -png -singlefile <name>.pdf <name>. They stay purely conceptual — no measured numbers (2026-09-02 rule).
• Data plots call apply_paper_style() from src/utils/paper_style.py before creating figures and take colours from its palette C: read feature / targets → C.read (green), write feature / FV / cue → C.write (blue), linear map / ridge → C.map (teal), baselines / unsteered → C.grey, counterfactual / other-task controls → C.accent (gold), secondary method variant → C.purple. Value labels via label_bars. No per-script rcParams, facecolors, or ad-hoc palettes.
• Restyling is style-only: titles, labels, numbers, filenames and output dirs never change with it.

2026-09-07 — Style-translation step 3 metric (user decisions)

• accuracy(family, style, k) = P(the completion uses the context's convention at cue k AND is a faithful, coherent English translation), one seeded T=1 sample per (text, style, k), n = 200. Style decision = registry classifier on the completion (word-internal cues prefixed by the already-generated word start); faithfulness/coherence = Gemini 2.5 Flash, one completion per call, told when the completion was cut by the 48-token cap. A completion that avoids the decision is inaccurate; its share is reported separately (unscorable.csv).

• Cue token (definition recorded 2026-09-07): the last token identical whichever style follows; k>=1 judged on the context as it reads; sentence families use the closing period (newline after English: for the first sentence); curly_quotes = one decision per quotation.

• Results bucket: results/style_translation/ (STYLE_TRANSLATION_RESULTS in paths.py); artifacts artifacts/style_translation/{prompts,rollouts}.

• percent_sign classifier (user decision 2026-09-07): 50 % (space before the sign) counts as the sign convention, per cent as spelled out. Rationale: the space is copied from the Spanish source typography, not a third English convention. properties.PercentSign.classify changed; step-3 percent_sign rollouts re-scored (134 decisions changed), plots regenerated. Angular quotes « » copied into curly_quotes completions stay unscorable.

2026-09-07 — style_free_text variation (user request): definitions

• Prompt = the English twin alone cut after cue token k (no Spanish, no header); cues recomputed on the header-free tokenisation; empty-context prompts start from <|endoftext|>. Judge = coherence/fluency only (no source to be faithful to — user decision); accuracy = convention AND coherent. Punctuation-only completions that close the sentence and carry a decision count as coherent (analysis rule; judge artefact without a reference). style_only_vs_translation.png is the like-for-like comparison with the translation run (judge-independent). Results bucket results/style_free_text/ (STYLE_FREE_TEXT_RESULTS), artifacts artifacts/style_free_text/. results/style_translation/ and its scripts unchanged.

2026-09-07 — Style-translation step 4: mean-difference steering at the cue token (user decisions)

• Steering vector: for family F, layer L (output of block L, L = 1..28), v_nat(L) = mean h_L(cue) over step-3 prompts whose completion was CORRECT in the nat context − the same mean in the alt context; "correct" = used the context's convention AND judged faithful/coherent (style_ok and judge.ok), all k = 0..4 pooled. v_alt = −v_nat. One vector per layer per style per family; no normalisation, no projection (capture_cues.py, artifacts/style_translation/steering/vectors/).
• Injection site: the k = 0 prompt (Spanish + English up to and including the FIRST cue token; identical for both styles), α·v added to the residual stream at layer L at the cue token only (last prompt position, prefill pass; generated tokens untouched — steer_hooks.CueSteer, unit-tested).
• Sweep: α ∈ {0, 0.5, 1, 2, 4} (α = 0 = unsteered, one grid point); screen layers {2,4,6,8,10,12,16,20,24} on 50 texts, style-only, 16-token completions; confirm the top-2 screen settings per (family, target) on all 200 texts with the FULL step-3 grading (style at the cue AND Gemini faithful/coherent, 48-token cap, capped flag). Final (L, α) = the confirmed setting with the higher full accuracy — a setting that forces the style but breaks the translation cannot win.
• Controls: unsteered α = 0 on the same 200 texts/seeds (plus step-3 k = 0 and k = 4 as references) and a counterfactual-family vector (another family's v, same style role, cyclic within the pod's shard) at the family's own top-1 (L, α).
• Results bucket results/style_translation/steering/; artifacts artifacts/style_translation/steering/{vectors,screen,confirm}.

2026-09-08 — Style-translation step 5: sparse head selection for FV-style convention steering (user decisions)

• Head vectors: raw mean out_proj-projected output of each of GPT-J's 448 attention heads at the cue token, over step-3 prompts whose completion used the ALT convention and was judged faithful (style_ok and judge.ok, all k pooled) — the function-vector recipe, not a nat/alt difference. Computed from the 120 fit texts only (my default, so held-out texts never enter the vectors).
• Sparse optimisation: one coefficient vector c ∈ [0,1]^448 SHARED across the 17 families (pooled prompts), L1 penalty λ, AdamW lr .01, batch 64, early stopping on a 20-text-per-family validation slice — the sandbox implementation src/sandbox/sparse_head_selection/train_sparse_heads.py reused unchanged except for a per-token MEAN loss wrapper (my default: all_caps/sentence_caps labels are whole first sentences and would otherwise dominate the pooled sum).
• Injection: v_F = Σ_h c_h C_alt[F,h] added at the cue token (last prompt position) of the k = 0 prompt at ONE layer; L ∈ {6, 9, 12, 16, 20} fitted separately, L* and λ* chosen by validation NLL (λ* = largest λ within 0.02 nats of the best).
• Golden completion / loss scope: the text_alt twin's continuation, teacher-forced, loss only on the tokens from cue+1 through the end of the alt style span (the style decision), not the whole sentence.
• Direction: alt only. Split: doc_id order 0..99 train / 100..119 validation / 120..199 test (80 per family; all graded numbers on the test texts).
• Grading as steps 3–4 (registry classifier ∧ Gemini judge). Arms: unsteered · sparse learned-c · sparse unweighted indicator sum (canonical FV construction) · other family's sparse vector (control) · step-4 residual vector and step-3 k = 4 on the same 80 texts.
• Results bucket results/style_translation/sparse_heads/; artifacts artifacts/style_translation/sparse_heads/{heads,points,train,eval,eval_lambda}.

2026-09-09 — Style-translation steering vectors: PAIRED selection is the default (user decision)

• Why: the step-4 vectors averaged over each pole's correct prompts independently. The pools differed in size (nat 2–3× alt where nat is the default) and in k (15–20 % k = 0 on the nat side, ≈0 % on the alt side, since the rare convention is almost never produced unprompted), so v_nat − v_alt could carry "short context / no prior decision" rather than the convention.
• Definition now: a (text, k) prompt pair enters both means only if the nat-context AND the alt-context completions were correct (style_ok and judge.ok for both twins). Both pools then hold the same texts at the same k (87–589 pairs per family; brit_t_past smallest). capture_cues.py default; --unpaired reproduces the old vectors.
• Scope: results/style_translation/steering/ is overwritten with the paired rerun (same screen/confirm/judge/analysis protocol). The unpaired results remain in git history (commit 17c52c5e for the figures, 146e6a6a/aac4cc6f for the failure-mode notes) and their artifacts in artifacts/style_translation/steering_unpaired/. results/style_translation/sparse_heads/ (step 5) was compared against the UNPAIRED residual steering; its comparison rows read artifacts/style_translation/steering/confirm and steering/best_config.csv, so re-running sparse_heads_analyze.py now compares against the paired vectors.

2026-09-10 — Style families in two categories (user decision)

• Lexically diverse (the convention applies across many different words, so the model must generalise a rule): us_uk, ise_ize, brit_t_past, contractions, num_words, ordinal_words.
• Lexically identical (one fixed marker or a pure formatting choice, the same every time): whilst, ampersand, percent_sign, em_dash, ellipsis, curly_quotes, quote_punct, double_space, oxford_comma, sentence_caps, all_caps. (whilst/while, &/and and %/percent are single fixed swaps, hence this group — user ruling.)
• Every per-family figure under results/style_translation/ uses family_groups.grouped_grid: left block = lexically diverse (blue tint), right block = lexically identical (warm tint), each with its own title. Lists live in src/sandbox/style_translation/family_groups.py.

2026-09-10 — Style-translation step 6: evidence tokens and read features (user decisions)

• Evidence tokens = the tokens of the English context that realise the convention, in the tokenizer's NATURAL segmentation (never a forced split): for one rendered instance, the style-bearing tokens from the first token where the twin and its single-instance flip differ (cue_idx + 1; the cue token is never included) to the last token starting inside the rendered span (trailing whitespace trimmed unless the span is all whitespace). Whole-word tokens stay whole (" learned"/" learnt", " colour"); where the tokenizer itself splits, only the differing pieces count ("'s" vs " is", "st" in "Among|st"); if the span is entirely shared with the other twin (nat "Among" vs "Among|st") the divergence token itself is the evidence (" these"), as for the absence poles (" and" for no serial comma). curly_quotes: opening + paired closing mark. evidence_tokens.py; per-family examples in results/style_translation/read_features/evidence_tokens.csv.
• Read feature r_pole(F, L) = mean over the 200 texts of the per-prompt average of hidden_states[L] over ALL evidence tokens of the 4 instances in the step-3 k = 4 translation prompt (no success filtering; both poles use the same texts). Per-instance-index means (1st…4th instance) are stored as well. All 28 layers captured; the 9 screened layers reported. capture_evidence.py → artifacts/style_translation/read_features/<family>.npz.
• Diagnostic (not a decision): cosine between the read difference r_nat − r_alt and the paired cue-token steering vector v_nat per layer.

2026-09-10 — Style-translation step 7: steering at the evidence tokens (user decisions)

• Read steering vector u_nat(L) = r_nat(L) − r_alt(L) (step-6 evidence-token means), u_alt = −u_nat; no normalisation. Note: ‖u‖ is 0.2–0.5 of the residual norm and 10–35× the cue-token write vector at early layers; α = 1 ≈ a full swap of the evidence-token state.
• Setting: the step-3 k = 3 translation prompts (3 rendered instances in context, model about to make the 4th choice), both context poles. α·u added at layer L at EVERY evidence token of the 3 instances (positions from the step-6 extraction; the cue token is untouched unless it is itself an evidence token — adjacent opportunities, 5.5 % of prompts, almost all all_caps where the 3rd sentence's final period is the 4th decision's cue). Directions: nat context → alt (u_alt), alt context → nat (u_nat). Sweep layers {2,4,6,8,10,12,16,20,24} × α ∈ {0.5,1,2,4}; screen 50 texts style-only → confirm top-2 on 200 texts with full grading, as in step 4.
• Metric: accuracy = TARGET convention at the 4th decision AND judge OK. Baseline = the unsteered k = 3 prompt scored toward the target (how often the model deviates from its context anyway). Reference = step-3 accuracy at k = 3 when the context genuinely was the target pole. Control (user choice) = another family's read vector at the same positions and (L, α).
• Results bucket results/style_translation/read_steer/; artifacts artifacts/style_translation/read_steer/{screen,confirm}; hook steer_hooks.PositionSteer.

2026-09-11 — Style-translation: ten more lexically diverse families (user decision)

• Families added (chosen by the user from a 20-item brainstorm; items judged overlapping or trivial were dropped): uk_vocab (truck/lorry), register (begin/commence), unit_abbr (5 km/5 kilometers), diacritics (cafe/café), latin_abbr (for example/e.g.), hyphen_compound (email/e-mail), flat_adverb (drive slowly/drive slow), irreg_past (dove/dived), latin_plural (indexes/indices), title_abbr (Doctor/Dr.). All ten are LEXICAL in family_groups.py (16 LEXICAL vs 11 FIXED). Registry src/sandbox/ext_styleprops/properties.py (27 properties), Spanish specs families.py (27).
• Scope: the new families get the Spanish→English translation setting only (no English-only style_free_text variant), then steps 3 (accuracy vs k), 4 (paired cue-token steering), 6 (read features at the evidence tokens) and 7 (evidence-token steering incl. the full 0–27 layer screen). Step 5 (sparse heads) is not rerun for them.
• House style = nat pole of all 27 families. The 17 original families' corpus, prompts and results stay as built on 2026-09-07 (17-family house style); their texts may contain the new families' alt forms ("repair", "railway", "well-being"), which does not affect their own decisions. pairs_check.py therefore reports ~100 "nat audit bad" texts per original family and 0 per new family — expected, not a defect.
• Detector design for ambiguous surfaces (flat_adverb): -ly adverbs are detected positionally (after any non-auxiliary word, clause-final: before punctuation / conjunction / preposition); bare flat forms only after a verbal word within 8 words with no copula in between, never after a measure ("30 cm deep") and never "even" (focus adverb). When the conservative alt detector under-counts on an alt twin, collect_english.build_twins accepts the pair if the classifier reads every rendered site as alt (and every nat site as nat). Rationale: the classifier at the cue is what scores the model; find_opps on running text must not corrupt house-style normalisation.
• Latin-plural lexicon: only pairs whose anglicised plural is standard American English; funguses / nucleuses / larvas / vertebras / alumnuses removed (Haiku judged them non-fluent).
• Haiku style_consistent is not gating for the English pass (unreliable for lexical families); the deterministic audit_nat is the style authority — same principle as the Spanish step (agent violations not used, audit is authority).
• Deterministic translator fixes: "that's," → "that is," before house-style normalisation (Gemini's rendering of «es decir»); the connective is exempt from the contractions renderer.
• Sentence cutter (2026-09-11, scoring fix): scoring.cut_sentence no longer treats the period of an abbreviation (Dr., Mr., Mrs., Ms., Prof., St., Mt., Ave., Rd., Blvd., Sr., Jr., No., vs., approx., e.g., i.e., single initials) as a sentence end. Before the fix 513 of title_abbr's 1,236 unscorable completions were the model's "Mr."/"Dr." cut off before the name. The ten new families' step-3 records were rescored from tail_raw (rescore_rollouts.py; title_abbr 680 tails / 502 decisions changed, unit_abbr 121 / 1, latin_abbr 70 / 0) and the changed tails re-judged. The 17 original families were NOT rescored (dry run: ≤ 7 tails and 2 decisions per 2,000 records would change) — their published results stand.

2026-09-11 — Style-translation on a second base model (user request)

• src/sandbox/style_translation/models.py is the model registry; --model <key> on cue_tokens.py, build_prompts.py, rollout.py, analyze.py. The default (gptj) keeps every existing path; any other model gets artifacts/style_translation/<key>/{cues,prompts,rollouts} and results/style_translation/<key>/. Cue tokens are tokeniser-specific and are recomputed per model; the GPT-J cues in dataset_files/style_translation/pairs/ stay the dataset's cues.
• qwen25_base = Qwen/Qwen2.5-7B (base, not Instruct), bf16, left padding, no BOS. Step 3 only (accuracy vs k) — steering/read-feature steps were not requested for it.

2026-09-14 — Qwen2.5-7B base convention pool for the read/write-feature and read→write-map study

• Pool = lexically diverse families whose k = 4 accuracy (convention ∧ faithful) is ≥ .30 for BOTH poles on the full 200-text corpus: 16 families — us_uk, ise_ize, contractions, num_words, ordinal_words, uk_vocab, register, diacritics, unit_abbr, latin_plural, title_abbr, zh_simp_trad, pt_acordo_eu, pt_br_eu, es_rae2010, ru_yo (results/style_translation/qwen25_base/pool.json). Fixed-marker families are excluded from both train and test. de_1996 and zh_tw_hk passed the cheap 45–60-text check but fell below on the full corpus → dropped.
• Feature gate before a family enters the map: steered accuracy's Wilson CI excludes the unsteered rate, in both directions.
• Generalisation test: leave-one-family-out over the pool + one fixed 2/3–1/3 split stratified by axis (British, Portuguese, Chinese, digits); within-axis vs cross-axis hold-outs reported separately. Coverage diagnostics are computed before steering.
• Multilingual corpora: English base text → Gemini adaptation in the natural pole with anchors → Gemini back-translation = source; Gemini verification (fluent, natural pole only, anchors, faithful); cap 200 per family.
• 2026-09-14 (later) — Feature gate variants: the strict gate (significant lift in BOTH directions) admits only 5 of 16 Qwen families because the natural pole of 7 families is at ceiling unsteered (.72–.94); the headroom-aware variant (a direction with unsteered ≥ .70 is exempt) admits 12. Both are reported; the 12-family pool is the main map result pending the user's ruling. Judge prompt: empty "so far" context (first opportunity opens the text) is now described as the beginning of the translation; 1,852 verdicts re-judged (Qwen steering confirm + step-3 rollouts).

2026-09-15 — Style-translation: coding-convention families (user decisions)

• Coding-language conventions are a third family group (family_groups.CODE, domain "code", src "Task"): the model writes code for a task description and the convention applies across many identifiers / literals / statements. 60 specs; 55 pass the cheap k=4 both-poles ≥ .30 check on 50 tasks and again on the full 200-task corpus (results/style_translation/qwen25_base/code_pool_full.json).
• Full corpora only for the 55 that passed at 50 tasks; steering confirm runs WITHOUT the counterfactual (other family's vector) control arm for now (steer_confirm.py --no_control).
• Cheap-corpus (50-task) rollouts and steering artefacts are kept aside under artifacts/style_translation/qwen25_base/{rollouts_cheap_k4,steering_cheap}/; the canonical code results are the full-corpus ones under results/style_translation/qwen25_base/{code,steering}/.

2026-09-15 — Style-translation: read feature for the coding-convention families (user decisions)

• Evidence tokens for code = the tokens of each earlier in-context opportunity that actually DIFFER between the nat and alt renderings (token-level difflib inside the opportunity span; shared content tokens inside merged spans such as ['rock', 'paper', …] are dropped). Whitespace-only evidence (indentation, blank lines, operator spaces) counts like any other token; no family excluded. Text families keep the 2026-09-10 whole-span rule (evidence_tokens.py, diff_only / n_shared_dropped per instance).
• The cue position is never steered in the read test: if the last evidence token of the last in-context instance is the cue (adjacent opportunities, ~0–1 % of prompts), that position is dropped (read_steer_screen.k_items); prompts left without evidence are skipped and counted.
• Read vector = paired mean difference of the evidence-token means over k = 4 prompts (documents with evidence in both poles); capture and evidence-steering screen over ALL layers 0–27; evidence steering tested on k = 1 AND k = 3 prompts (--k_ctx; artifacts screen_k1 / confirm_k1, results read_steer_k1/); no control arm (as on the write side).
• Gate for "the read feature works": steered accuracy toward the target ≥ 50 % of the flipped reference (step-3 accuracy at the same k of prompts whose context genuinely shows the target) — reported as reach / pass_50 in best_config.csv together with the Wilson-CI-vs- unsteered flag; the user judges the final results and decides the gate. Read-out check = cosine per layer between the read vector and the write vector (same layer and L24); no linear probe.
• read_features_analyze.py, read_steer_analyze.py, read_steer_layers_analyze.py take --tag/--families so code results live under results/style_translation/qwen25_base/code/{read_features,read_steer,read_steer_k1}/, never mixed into the text buckets.
• 2026-09-15 (later) — Shared read layer for code = L8 (user decision; write layer stays L24). Screen evidence: the mean curve over the 55 families peaks at L6–L8 with α = 4; at L8 with the best α per family the mean target rate is .62 vs .71 for each family's own best layer at k = 3 (loss .085; .55 vs .65 at k = 1), and α = 4 is the best shared α at L8 (nat→alt .54, alt→nat .63 at k = 3). The judged confirm arms were run at each family's top-2 screen cells, so a judged L8-only number needs a small extra confirm run.
• 2026-09-15 (later) — Read→write map on the code pool (user decisions): all 55 families; ONE fixed random split 2/3–1/3 stratified by convention category (seed 2026, code/read_write_map/fixed_split.json, committed before fitting), no LOFO; per-prompt ridge from the evidence-token read activation (L8 main, sweep L5–L10) to the cue-token write activation (L24); judged by held-out geometry only (convention-vector cosine vs the shuffled-pairing control and the mean-training-vector baseline, prompt-level R², coverage diagnostics). Steering with predicted vectors is a separate task.
• 2026-09-15 (later) — Working read layer for the code map = L10 (user decision): per-prompt held-out centroid R² .340 at L10 vs .333 at L8 (80/20 split; spread over L5–L10 is .03). L10 is the working choice for improving the map; the other layers get re-optimised once an improvement is found. L8 stays the reported read-steering layer.
• 2026-09-16 — k = 8 probe for code (user request): k = 8 only (no k = 5–7); k = 8 prompt = 8 in-context opportunities, decision at the 9th; documents with ≥ 9 opportunities only; headline = PAIRED k = 4 vs k = 8 on the same documents. k = 8 items live in the same prompt and rollout files (append mode) so write vectors can pool or select k = 8; the canonical write vectors remain the k = 0..4 ones (steering/vectors/); any k = 8 vector goes to steering/vectors_k8/ via capture_cues.py --ks 8 --out_tag k8.

2026-09-16 — Code conventions: free-opportunity rule (user finding + decision)

• Finding: in the identifier-naming families a large share of decisions were FORCED by the code rather than by the convention — once an identifier is defined, later mentions must reuse its spelling for the code to work (absolute_value … current_number = absolute|_value); self must match the parameter name; lines inside a block must keep its indentation; the task text can pin a function name. Forced share of the k = 4 decision: py_self_name .99, py_loop_vars .63, py_snake_camel .59, js_camel_snake .56, py_abbrev .56, js_hungarian .41, py_bool_prefix .39, py_class_naming .35 (py_private .04, py_const_naming .02). Forced decisions were far easier (e.g. py_class_naming k = 4 nat .26 free vs .82 forced), so k-shot accuracy and some steering numbers were inflated. Quotes, keywords, operators, literals and the other formatting families are not affected (either rendering stays legal).
• Decision: rebuild the 12 affected families (9 identifier families + py_self_name + py_indent + py_tabs) so that every opportunity is a genuine choice (code_free.py): identifier families — no identifier of the span, in either rendering, may occur earlier in the code or be pinned by the task text (backticks / call form); py_loop_vars — only the binding in a for header; py_self_name — only the self parameter of a def; py_indent / py_tabs — only the first line of a new block. Pairs keep only free opportunities (≥ 5, 5th before 75 %); documents that fall short are replaced by newly generated twins whose generation hint asks for many fresh choice points (code_build.EXTRA_HINT, --target 200). The old data for these families is archived under dataset_files/style_translation/pairs_v1_forced/ and artifacts/style_translation/qwen25_base/v1_forced/; every downstream step (step 3, write and read features, map) is rerun for them.
• 2026-09-16 (later) — After the free-opportunity rebuild the both-poles ≥ .30 cutoff drops py_bool_prefix (alt .12) and py_indent (2-space .24); the code pool is now 53 families (code_pool_full.json, old 55-list kept as pool_v1_forced). js_hungarian (nat .66 → .38) and py_abbrev (nat .72 → .40) lost most but stay. The k = 8 numbers in code/k8/ predate the rebuild for the 12 affected families.

2026-09-17 — Whitespace-only code families: alternative twin derived by rule, not by LLM rewrite

• py_indent, py_tabs, blank_lines, operator_spaces, comma_space, line_wrap: text_alt = code_whitespace_alt.transform(family, text_nat) (tokenizer-aware; strings, docstrings and comments are never edited; both twins must have the same AST). The Gemini rewrite had slipped content edits into 11 of ~1,100 documents and had "applied" the convention inside prose (docstrings, commented-out code), which the rule refuses — so an opportunity in these families is always a genuine code-formatting choice.
• line_wrap: joining a wrapped call / def / literal also drops Black's trailing comma (a 1-tuple's or subscript's comma is kept) — the one non-whitespace difference allowed in these families (user-reviewed as legitimate, bug-4 explainer t109).
• Records carry alt_rule: true; code_build.build_one applies the rule for these families instead of the REWRITE prompt.

2026-09-17 — Free-opportunity rule, content families (bug 5)

• early_return, py_with_open, py_ternary: an opportunity counts only if its two renderings differ beyond whitespace. A block choice (else:, with, an if/else block for a ternary) forces the indentation of every following line; those diff spans are kept in opps_all but never counted, never cued, never used for prompts or features.
• A whitespace-only difference before the first real choice is a rewrite slip, not a convention: it takes the natural rendering (code_free.harmonise_leading_ws), so the k = 0 prompts of the two poles are byte-identical.
• Generator guard: a Python twin that does not parse is rejected (code_build.valid_python; py2_print / py2_except alternatives exempt).

2026-09-17 — Code pairs: validity guard (bug 6) and bash_subst alternative by rule (bug 7)

• Every Python twin must parse (code_build.valid_python, py2_print / py2_except alternatives exempt); the generator rejects otherwise. Style hints must say what stays untouched (py_paren_if, trailing_commas, comment_case EXTRA_HINTs) — Gemini over-applies a bare hint.
• bash_subst: text_alt = code_subst_alt.convert(text_nat); nested substitutions use Bash's escaping (`…` inner pairs, backslashes before $ \ ` doubled per level). Acceptance = independent parse-back to the same substitution tree + bash -n on both twins.

2026-09-17 — Identifier families: rename-collision guard (bug 8)

• A rename x → y in the alternative twin is invalid when y already exists as a standalone name in the natural code (code_free.rename_collisions, token-aware); such pairs are rejected by consistent_twins and regenerated. Generation hints for py_class_naming / py_bool_prefix forbid the collision up front. opps_all (the full diff) is mandatory on every free-filtered record — never re-filter from the filtered list.

2026-09-17 — Paired symbols (bug 10) and the 200-per-family corpus

• A closing bracket or closing quote is never an opportunity (code_free.closing_symbol); the free-opportunity machinery (opps_all full diff, opps counted) now applies to every code family. Every family has exactly 200 documents (css_shorthand pending its follow-up run).
• The builder's re-filter (apply_free) may only remove a pass; generation-time guards (parsers, degenerate, shared fraction) are final.

2026-09-17 — Two-part constructs: only the opener is an opportunity (bug 11)

• bash_subst ($( … ) vs backticks) and py_with_open (with open(…) as f: vs f = open(…) … f.close()): the second half is guaranteed by the first, so only the opener is counted (code_free.OPENER_ONLY). Diagnostic to remember: an accuracy / unscorable curve that alternates with the parity of k means the opportunity list interleaves free and forced decisions.

2026-09-17 — One decision per natural-twin line (bug 12)

• rust_question, c_comment_style, py_ternary, py_enumerate: a construct that sits on one line of the natural twin is ONE opportunity — the first counted span on that line; later spans on the line follow from it (code_free.one_per_line). code_free.counted(fam, rec) is the single definition of the counted list (per-span free rule, then the line rule); sweeps must use it.
• Solution length guidance is per family (code_build.LENGTH), so families that need ≥ 8 line-level constructs get 30–50 lines.

2026-09-17 — Rollout scoring: only what the completion introduces may decide

• For first-mention (identifier) families a completion's convention is read from the first NEW name it introduces; names already in the code so far or pinned by the task are forced reuses and never decide (code_scoring.decide_code). Same principle as the free-opportunity rule, applied to scoring. hex_constants: first integer literal. py_abbrev / py_bool_prefix: LLM-labelled lexicon of new identifiers (Claude Sonnet 5).
• Any change to a scorer must be validated against independent LLM labels on a random sample (validate_scoring.py pattern) and reported as agreement before / after; the log-prob margin at the cue is the reference measure that no classifier can bias.

2026-09-17 — One construct = one decision, for every family (bug 13)

• A counted opportunity must be the FIRST diff span of its construct: no earlier content-or-eligible span on the same line of either twin (docstring_style: per docstring; sql_join_style: per statement). Name / self / indentation families keep their own unit (new name, new block).
• The integrity sweep now includes two_counted_on_a_line; any new family must pass it. When one family shows a forced-decision artefact, audit ALL families for the same class at once (structural check + log-prob first-vs-later comparison) instead of fixing symptoms one by one.

2026-09-17 — Write-feature steering for code conventions: definitions and selection rule

• Steering vector per family and layer: v = μ_nat − μ_alt, where μ is the unpaired mean cue-token activation (output of block L) over k ∈ {3, 4} prompts whose sampled completion followed the context's convention AND passed the judge, built from the 150 training documents only. Held-out split is deterministic: write_split.heldout(doc_id) = crc32 % 4 == 0 (~50 docs); held-out 0-shot prompts are the only test set.
• Intervention: add ±α·v at the cue token (last prompt token) of the 0-shot prompt during prefill, one layer at a time; nothing at generated tokens. α is a multiple of the raw difference. "Every other layer" = blocks 2, 4, …, 26; block 28 excluded (post-norm mean does not match the hook site and cannot affect later tokens).
• Grading identical to unsteered 0-shot rollouts (T = 1 sample, 48 tokens, cut_code, context-aware decision, Gemini judge); success = target convention AND judge OK. First-token log-prob margin kept as the classifier-free check.
• Hyperparameters are chosen ONCE on 10 random pool families and shared by all 56: the cell maximising mean success over families and both directions (ties → smaller α, then earlier layer). Result: layer 26, α = 2 (pending user approval). Late-layer-only effect and the first-token-not-convention behaviour are recorded in WORKLOG; do not present L26 steering as evidence about mid-layer write features.

2026-09-18 — Write steering, full pool: four cells reported, none selected

• All 56 pool families are steered at (L24, L26) × (α 2, 4) on the first 40 held-out documents (sorted doc_id; write_sweep.py --limit 40), each towards natural and towards alternative; the 10 sweep families are reused and cut to the same 40 documents. No single cell is selected; full56/by_cell.csv reports the four means side by side with the unsteered baseline and the k = 4 in-context accuracy.
• Sweep lesson carried over: only layers 24/26 respond and the effect is largely a first-token push; families whose convention needs follow-through, or whose construct the 0-shot model never writes, do not respond. Do not read these numbers as evidence about mid-layer write features.

2026-09-18 — Write feature for code conventions = layer 24, α = 2 (USER DECISION)

• The write feature of a code-convention family is the cue-token mean-difference vector at the output of block 24, v = μ_nat − μ_alt (μ from k ∈ {3, 4} correct completions on the 150 training documents), applied as ±2·v at the cue token of a 0-shot prompt. This is the go-to definition for all further code-convention work (read→write maps, comparisons, write-ups); the other three cells in full56/ stay as reported context. Headline numbers at this setting over 56 families: → natural .56 (unsteered .40), → alternative .43 (unsteered .13), judge OK .84; k = 4 in context .74 / .69.

2026-09-18 — Bugs 14/15: cues must sit where the convention is decided (USER DECISIONS)

• Comment families (comment_language, comment_case), bug 14: one opportunity per comment unit. A # comment is a unit; of a docstring only its FIRST line with text is a unit (later lines are free); a # line that continues the previous # comment's sentence (previous line without terminal punctuation, this comment starting lowercase) is free. A unit counts only if its first word differs between the poles, so the cue is the comment opener (# or the docstring line's indentation). String literals are never units. First words that differ only by a suffix (Increment|ar) stay counted. Residuals accepted as they are: 18 docs (3 below 5 opportunities, 15 whose leading unit is free so their k = 0 prompts differ). comment_case keeps its docstring units (few, 51).
• Audit of the other families: the token diff merges the closer of one construct with the opener of the next across a shared line break, so cues land before a forced closer or inside a construct that already showed its style. Four fixes were proposed: 1 split raw spans at shared line breaks — NOT adopted in general (in k-shot prompts a closer still tests whether the convention is continued sensibly); flagged for a possible revisit; applied only inside the families that fixes 2–4 touch, where needed. 2 a content span blocks every line it covers (gated to sql_keyword_case, php_array, py_literal_ctor [with the split], py_join_concat, c_comment_style) — ADOPTED. 3 closers followed by ,/; are forced closers — ADOPTED, universal. 4 line-level alignment for the block-restructuring families py_ternary, py_comprehension, rust_question, js_arrow, py_with_open (code_line_align.line_opportunities: each replaced block of lines = one construct, cue at the first divergence; blocks with several adjacent constructs are split one per construct; forced closers/rewrite artefacts free) — ADOPTED with the split. Families that only show the fix-1 pattern (py2_print, py_quotes, js_quotes, py_fstring, bash_test, line_wrap) are left as they are.
• sql_keyword_case: the LLM-written alternative twin lower-cased identifiers, string literals and comments too; the alternative twin is rebuilt BY RULE from the natural one (code_sql_alt.py), lower-casing only SQL keywords, as for the whitespace families.
• Documents that fall below 5 opportunities, whose alternative rewrite violates the family rule, or whose k = 0 prompts differ are regenerated from their own tasks under all guards; nothing else is regenerated. Changed families are re-sampled for step 3 and for write steering (L24, α = 2 plus the other three cells for the full56 table).
• Evidence-token definition agreed so far: the token right after each demonstration's cue, plus every other demonstration token that differs between the two poles' renderings (opening AND closing quotation marks; all comment words; the closing position of an absence pole such as : vs ):). Scope of the alignment (whole prompt vs demonstrations only) and the treatment of tokens that differ only by a leading space after an inserted symbol: still open.
• 2026-09-18 additions after the bug 14/15 re-sample: write_sweep.py skips held-out documents whose k = 0 prompts differ between poles (accepted comment_language residuals) instead of asserting; js_arrow generation hint now asks for ≥ 10 arrow functions (30–45 lines) because the line-level rule counts ONE opportunity per arrow; 41 documents whose 5th opportunity sits past 75 % of the document are accepted as a residual (user decision); the pool stays 56/60 after the re-sample.

2026-09-18 — Evidence tokens for the code-convention families (USER DECISION, complete)

Evidence tokens of a k-shot prompt = (1) every token before the query cue is in scope (demonstrations, free occurrences between them, and anything after the last demonstration); (2) the token right after each demonstration's cue, plus every other token that differs between the two poles' renderings (token-level alignment of the twins; absence poles get their closing position, e.g. : vs ):; free repeated uses such as later self/this count); (3) SECTION RULE for comment_language, comment_case, docstring_style: every token of each comment / docstring before the query cue, opener to closer, shared tokens included (the query's own, just-opened section excluded); (4) a word that differs only by the whitespace in front of it (an inserted symbol took its space: (idx vs idx) is NOT evidence, EXCEPT in comma_space, operator_spaces, line_wrap and py_tabs where that whitespace is the convention; pure whitespace tokens always count; (5) cue tokens (demonstration and query) and the task header are never evidence. Identical words inside a rewritten region are matched as equal by the alignment and are not evidence (an earlier claim to the contrary was a measurement error). Evidence sets range from 3 tokens (js_strict_eq, r_assignment) to ~150 (docstring_style) per k = 3 prompt; the read feature is the mean over them, so per-family results must be read with that in mind.

2026-09-18 — Read feature and read steering (definitions; USER answers)

• Read feature of a family, pole and layer = mean over prompts of the per-prompt mean activation over the prompt's evidence tokens (code_evidence.py rule), at the output of block L (1..28, layer 0 stored too); prompts = exactly the write feature's set (k ∈ {3, 4}, training documents, completion correct, unpaired). Read vector r_L = mean_nat − mean_alt.
• Read steering test = held-out k = 3 prompts whose demonstrations are in the WRONG style, ±α·r_L added at EVERY evidence token of the prompt at layer L (prefill only, nothing at the query cue or generated tokens); success = target convention AND judge OK; baseline = the same prompts unsteered; ceiling = the right-style k = 3 accuracy. Sweep families = the 10 write-sweep families. Selection rule as for the write sweep (one shared cell, max mean success). Result: L8, α = 4 (pending approval; α not saturated).
• 2026-09-18 USER DECISION: read-steering setting = layer 8, α = 4 (top-5 cells L8–L18 at α4 tied within 1.6 points; layer choice within 6–18 is arbitrary, α still rising at 4 — a wider α range was offered and declined for now). Full-pool run done at that setting; results in results/code_styles/read_steering/full56/.

2026-09-19 — Padding in generated documents (USER DECISIONS)

• Generated code must not contain padding: filler / placeholder / dummy / illustrative lines, discarded results, comments announcing examples or counting occurrences, stacked negations, meta-comments naming the convention. code_build.PADDING_PATTERNS + padding_lines() is a builder guard and the sweep check padding_line (code_pairs_check.py); # Example usage blocks and not a or not b are NOT padding.
• For py_not_in, rust_question, py_is_none, float_literals the numeric occurrence hint is REMOVED ("use the construct wherever the logic calls for it"; a realistic module with a driver is allowed; LENGTH 40–80; 5th opportunity < 85 %); all 800 documents were regenerated with Claude Opus 5 and accepted only if TWO independent reviewers (Opus 5, GPT-5; 14-item checklist; strict JSON with quoted lines) both pass, with a feedback-repair loop (reviewer issues fed back to the generator; ~1.6× the per-attempt yield). The ≥ 8 hint stays for the other families. The other 52 pool families still trip padding_line in 2,281 documents — open.
• Task pools whose specs do not naturally need the construct cannot yield honest documents with many occurrences; the honest count of not in is ~5 per document. Prefer tasks that inherently need the construct when new families are built.

2026-09-23 — code-style corpus: GPT-5 is the sole document judge; regen8 pipeline is the standard

• Reviewer: GPT-5 only (Opus 5 + GPT-5 with an AND rule was too strict; user decision). Automatic guards carry the padding burden: padding_lines, leak_lines, pyflakes unused/undefined names, JS unused declarations, sandboxed run of the natural twin, string-literal equality (code_static_checks.py).
• Alternative twins come from exact, validated rules wherever one exists (code_rule_alt.py + code_rules_a/b/c.py, 37 families); a declining rule falls back to the LLM rewrite and is counted. Tasks come from the designed pools (tasks_designed_<family>.json); the generic pools are retired for code.
• Lesson: task reservations across parallel drivers must exclude every accepted-but-unmerged id (regen8 collision bug).

2026-09-23 — js_hungarian dropped from the code-style pool (USER DECISION)

Hungarian-notation prefixes encode a variable's type, so the alternative twin cannot be derived by an exact rule (the only naming family where that holds); model rewrites leaked prefixes into string literals and task-pinned parameters stayed unprefixed, so no regeneration passed review (0 / 9, then 0 / 20). The family stays in the dataset and the artefacts but is excluded from code_pool.json by code_cutoff.py --exclude js_hungarian (default) and is not reported as a data issue any more. Pool = 53 / 60.