Experiment equipment？

[Bing-a-ling7]

Hello, great work! I’m trying to reproduce your paper and was wondering: what kind of GPU did you use to train the SAE, and how long did the training take?

[avgalichin]

Hello!

Thank you for the interest in our work.

To train the SAE, we have used 1 H100, 80GB and the training took approximately 30 hours. But I believe you can actually train the SAE with less GPU memory or lower-end GPUs (e.g. A100) by adjusting the train_batch_size_tokens, n_batches_in_buffer, and store_batch_size_prompts accordingly. Although at the cost of increased training time. You can also reduce the context_size to accelerate the training.

[Bing-a-ling7]

To train the SAE, we have used 1 H100, 80GB and the training took approximately 30 hours. But I believe you can actually train the SAE with less GPU memory or lower-end GPUs (e.g. A100) by adjusting the train_batch_size_tokens, n_batches_in_buffer, and store_batch_size_prompts accordingly. Although at the cost of increased training time. You can also reduce the context_size to accelerate the training.

thank you! I will try it