Code for pre-training a GPT-2 model on (eight) NVIDIA DGX B200 GPUs and short tutorial on the topic. Uses Torch and HF Transformers. It can pre-train GPT-2 Small on 32 GB of data in around 2.5 hours. It handles dataset tokenization too.