Distill or prune model to save training time

[johnml1135]

If we can distill or prune the NLLB-200 shortly after starting fine tuning, we may be able to dramatically reduce (50% or more) the training and inferencing time needed. It could even do something like this:

• Take the 3.3GB model and train for 1000 steps on the 2x A100's. Prune and save.
• Load the model on a single A100 and finish training and inferencing.

[johnml1135]

A few resources on pruning:

• https://towardsdatascience.com/head-pruning-in-transformer-models-ec222ca9ece7
• https://towardsdatascience.com/how-to-prune-neural-networks-with-pytorch-ebef60316b91
• https://arxiv.org/pdf/1606.09274.pdf
• https://aclanthology.org/2020.emnlp-main.211.pdf

[ddaspit]

We can prune a model during training using Optimum.