.

README.md

@@ -101,7 +101,7 @@ yeah, those guys doing free labor who everyone constantly musters all of their i
 ### Speedrun rules
 
 1. Must not modify the train or validation data pipelines (except to change batch size & seqlen; i.e., just the order of the tokens can't be changed).
-2. Must use ≤ 124M active parameters per token.
+2. Must use ≤ 124M active parameters per token (MoE is OK; & untied embedding matrix only contributes hidden_dim active params).
 3. Must attain ≤ 3.28 val loss. A tasteful number would be 3.278 so that [this doesn't happen](./records/110924_Replicateleloykun/1621af10-aa0c-42af-bf54-8a773c63a2af.txt#L3780).
 
 Other than that, go crazy! Anything is fair game