Python notebooks and scripts to do experiments with toy transformer models. Results are summarised in blog-style posts at https://efarrell1.github.io/.
transformer_tests.py contains a bunch of custom functions, drawing heavily from Neel Nanda's TransformerLens tutorials and these excellent tutorials by Callum McDougall.
The code in induction_head_recency_bias.ipynb produces the results discussed in https://efarrell1.github.io/posts/recency-bias-induction-head/.
The code in generalised_induction_sequences.ipynb produces the results discussed in https://efarrell1.github.io/posts/recency-bias-induction-head/.
The code in previous-token-head.ipynb approximately reproduces the results discussed in https://efarrell1.github.io/posts/previous-token-head/.
The code in Positional_Embeddings.ipynb approximately reproduces the results in https://efarrell1.github.io/posts/pos-embeddings-2layer/