About fine-tuning on CLEVR-Humans

[dandelin]

Hello again!

If you don't mind, I have one more question for detailed procedure of fine-tuning the CLEVR-Humans dataset.

I was able to reproduce 12-step MAC's accuracy (98.9%) using Pytorch, but failed to reproduce Humans after FT (result was 76.6%, lower than paper's 81.5%).

My fine-tuning was done by (1) load fully trained model on CLEVR, (2) initialize new words' embedding vectors just as original words, (3) re-training the model on CLEVR-Humans train dataset ONLY following original model's learning schedule.

It seems your fine-tuning code trains the model on mixture CLEVR and CLEVR-Humans train dataset rather than using only CLEVR-Humans train dataset. (sorry if I misread again 😢) So I'm guessing that this difference might be the reason.

Since using the mixture of both dataset will take longer than just using CLEVR-Humans, I'm opening the issue thinking you might encountered the same problem and could help me out.

Thanks!

[dorarad]

Hi! thanks again for the interest! :)
I actually have tried several different learning schedules, including both mixtures and first training on clevr and then fine-tuning, but not waiting too many epochs of training on clevr before starting fine-tuning such that weights won't get too "fixed" at their optimal configuration for clevr, so I varied the number of epochs i trained on clevr before moving to fine-tuning.
Another important factor I played with a lot then was the word embeddings, I tried several different versions like both random vectors and glove, and also did some scaling to the norm (e.g. normalizing word vector lengths). It had quite impact actually for clevr humans, probably because of the relatively small amount of clevr-humans training data.

I'm happy to look more into the details of learning scheduling and the settings but I will probably be able to do that only in two weeks, so I hope that timing is good and will be happy to follow-up with more information then! :)

[dandelin]

Thanks for the reply!
I’ll wait for the detailed learning schedule then (NeurIPS deadline maybe? :o).

[dorarad]

yep! :)