GPT to Llama (rasbt#368)

* GPT to Llama * fix urls
ahnjj · Sep 23, 2024 · 0467c82 · 0467c82
1 parent dc1b1a0
commit 0467c82
Show file tree

Hide file tree

Showing 7 changed files with 1,644 additions and 0 deletions.
diff --git a/.gitignore b/.gitignore
@@ -34,6 +34,8 @@ ch05/01_main-chapter-code/model.pth
 ch05/01_main-chapter-code/model_and_optimizer.pth
 ch05/03_bonus_pretraining_on_gutenberg/model_checkpoints
 ch05/06_user_interface/gpt2
+ch05/07_gpt_to_llama/models--meta-llama--Llama-2-7b
+ch05/07_gpt_to_llama/models--meta-llama--Llama-2-7b-chat
 
 ch06/01_main-chapter-code/gpt2
 ch06/02_bonus_additional-experiments/gpt2

diff --git a/README.md b/README.md
@@ -116,6 +116,7 @@ Several folders contain optional materials as a bonus for interested readers:
   - [Adding Bells and Whistles to the Training Loop](ch05/04_learning_rate_schedulers)
   - [Optimizing Hyperparameters for Pretraining](ch05/05_bonus_hparam_tuning)
   - [Building a User Interface to Interact With the Pretrained LLM](ch05/06_user_interface)
+  - [Converting GPT to Llama](ch05/07_gpt_to_llama)
 - **Chapter 6:**
   - [Additional experiments finetuning different layers and using larger models](ch06/02_bonus_additional-experiments)
   - [Finetuning different models on 50k IMDB movie review dataset](ch06/03_bonus_imdb-classification)

diff --git a/ch05/07_gpt_to_llama/README.md b/ch05/07_gpt_to_llama/README.md
@@ -0,0 +1,7 @@
+# Converting GPT to Llama
+
+
+
+This folder contains code for converting the GPT implementation from chapter 4 and 5 to Meta AI's Llama architecture:
+
+- [converting-gpt-to-llama2.ipynb](converting-gpt-to-llama2.ipynb): contains code to convert GPT to Llama 2 7B step by step and loads pretrained weights from Meta AI