add deterministic flag + functionality

[alexander-soare]

Follow up on discussion #845 (comment)

[rwightman]

@alexander-soare that still won't be deterministic no? torch.use_deterministic_algorithms(True) needs to be set as it has broader scope than cudnn flag. Also not sure if the benchmark mode needs to be disabled as well of uf use_deterministic_algorithms overrides that too...

[alexander-soare]

@rwightman hmm yeah I must admit I'm not totally sure about this one. Empirically, it does what it's supposed to do. And using torch.use_deterministic_algorithms(True) raises issues with some algorithms not having determinism implemented (strange when considering my empirical success). I haven't dug too deeply into the docs to try to understand exactly what's going on. Same thing re. your benchmark comment - I tried without and it just works.

Happy to park this one until/if I get around to fully understanding what's going on.

[rwightman]

@alexander-soare deterministic is always such a bother :) so in your trials just the cudnn flag seemed to result in reproducible runs where as without it was different? It's definitely not a straightforward yay/nay... and yeah, the full flag it can break some models since not all ops have deterministic impl. I'll think about it, maybe make the deterministic setting a string/enum so one can do --deterministic cudnn, --deterministic full, etc

.. adding the flag and then not actually having it fully deterministic seems misleading, hah

[alexander-soare]

@rwightman

so in your trials just the cudnn flag seemed to result in reproducible runs where as without it was different?

Correct.

And yeah makes sense to drop this for now. Besides, it's easy enough to diy for whoever wants it.