CogView Codebase Deep Dive

[afiaka87]

Let's explore the CogView codebase which was released today. It's impressive work and the code is quite clean (at least the tiny bit I've read so far).

https://github.com/THUDM/CogView

I've made a semi-related issue regarding the need for a big refactor of the codebase here:
#309

[afiaka87]

The training code is here:

https://github.com/THUDM/CogView/blob/169a0234990d4507ab5ec8e70b15831d8d5b3dd3/pretrain_gpt2.py

And is surprisingly simple.

[robvanvolt]

My first few thoughts reading the code / their utilities.py:

• I like the idea of "outsourcing" the logging, so the training can be kept clear and concise
• timer implementation (time to finish epoch / time to finish training)
• looking into the requirements / tensorboard implementation, I wonder if they benefitted any more than they would with wandb

To be continued...

[afiaka87]

Agreed - @janEbert I haven't vetted their distributed implementation but there certainly seem to be some insights from them that we could use.