don't dropout in eval mode

[YassineYousfi]

To give the model more chances to figure out if it's being trained or deployed :) (c.f. https://twitter.com/karpathy/status/1635049541534879745)

[karpathy]

Whoa hold on this is an actual bug in my implemtnation right? Basically we are applying dropout at evaluation time?

[YassineYousfi]

Yes. I only noticed it recently while training some models with dropout.

[karpathy]

😱 😱 😱
:'(
... shook. calling functional directly considered dangerous
ty

[prshnthrv]

I see that if I use different dropout values and run eval using the gpt2 model, I get different results. Any idea why?
For example,
dropout = 0 and I run
python train.py config/eval_gpt2.py
I get a val_loss of 3.09, which is close to what you get.
If I set dropout = 0.1 and run eval again, I get a val_loss of 3.49
With dropout = 0.2, val_loss is 4.23.
Any idea why? I stepped through and indeed the model is in eval() mode and dropout should not have any impact during evaluation, correct? What am I missing?

[YassineYousfi]

You mean after this PR was merged?
There should not be any difference in the first call of estimate_loss() after this PR was merged.

[prshnthrv]

I think before this PR. This PR only changes if we are using the flash attention module. In my setup, I am not using flash attention. So this fix should not have any impact on my evaluation.

[YassineYousfi]

hmmm, I don't think I can reproduce. You should open an issue with your setup + script to reproduce

[prshnthrv]

If you run
$ python train.py config/eval_gpt2.py
but change the dropout value in https://github.com/karpathy/nanoGPT/blob/master/train.py#L55 to 0.1 or 0.2, the val_loss is different as I mentioned earlier.
I can open another issue.

[YassineYousfi]

Yes I just did that on current master (with the shakespeare dataset, with and without flash attention) and the losses were the same.

[prshnthrv]

Interesting. I’m using openwebtext. I can try with Shakespeare as well. But was trying to match the owt numbers specified.