What is the reasoning for no dropout?

[luchris429]

recurrent-memory-transformer-pytorch/recurrent_memory_transformer_pytorch/recurrent_memory_transformer.py

Line 135 in d45ef72

dropout = 0.,

[lucidrains]

@luchris429 for a while, there was this new fad of not doing dropouts for LLMs, simply because OpenAI stopped doing them as they scaled up

now i see papers like this and realize nobody really knows what they are doing. yea, let's get the dropouts back in there.

[luchris429]

ahh I did not know. thanks for the context!