Is running the notebook Necessary?

[AminTaheri23]

When executing python autogen.py add_example nlp/script_name is it necessary to run the whole notebook again? Some of the use cases for Transfer learning (for instance BERT and GPT2) are quite slow and have an enormous processing hunger. this will make running the add_example command harder.

[fchollet]

Some of the use cases for Transfer learning (for instance BERT and GPT2) are quite slow and have an enormous processing hunger.

Then make it smaller / lighter. We need to run the notebooks regularly as a form of integration testing. There's no way around it.

Note that you don't have to train to convergence. You could just train for one epoch. Or on 1000 samples. And leave convergence as an exercise for the reader.

Also note that the exact md and ipynb files that you commit may get regenerated by us at any time. The py is the only source of truth. The md and ipynb are for display only.

[fchollet]

As a general rule, an example tutobook should run under 30min on GPU.