Principal-Spectral-Regularization-For-LLM-Training

Table of Contents

• Setup
• Download Dataset
• LLaMA Pretraining

Setup

To set up the environment for this project, follow these steps:

1. Clone the repository:

   git clone https://github.com/xie-lab-ml/Principal-Spectral-Regularization-For-LLM-Training.git
   cd Principal-Spectral-Regularization-For-LLM-Training

2. Create a virtual environment with Conda:

   conda create -n llm-psr python=3.10
   conda activate llm-psr

3. Install dependencies:

   pip install -r requirements.txt

Download Dataset

To download the processed 100k-doc training sample dataset, run the following command:

# Download llama model data
wget https://bj.bcebos.com/paddlenlp/models/transformers/llama/data/llama_openwebtext_100k.bin
wget https://bj.bcebos.com/paddlenlp/models/transformers/llama/data/llama_openwebtext_100k.idx

LLaMA Pretraining

To run the default pretraining experiment on LLaMA models, run the following command to invoke the script:

python -u -m paddle.distributed.launch --gpus 0,1,2,3,4,5,6,7 run_pretrain.py config/llama/pretrain_argument.json