Default MIOPEN_FIND_MODE=FAST for amd instead of disabling cudnn

[alexheretic]

Defaulting MIOPEN_FIND_MODE=FAST, when this env var is not already set, may fix warmup slowness on AMD without the downside of disabling cudnn fully which has negative perf effects on rocm 6.4.

This is also easy for the user to opt out of, since they can simply set MIOPEN_FIND_MODE themselves.

@comfyanonymous can you check if this works as well as disabling cudnn for your test scenarios?

Resolves #10447

[asagi4]

This doesn't really improve performance so much as it reduces initial latency.

Running MIOPEN_FIND_MODE=hybrid (the default) will have a slow initial run, but the subsequent runs should be faster after your MIOpen database has been populated.

I think setting fast by default if it's unset might still be a good idea though, but you might want to give a hint to people about trying hybrid as well if they want to optimize performance.

[alexheretic]

This doesn't really improve performance so much as it reduces initial latency.

Running MIOPEN_FIND_MODE=hybrid (the default) will have a slow initial run, but the subsequent runs should be faster > after your MIOpen database has been populated.

I think setting fast by default if it's unset might still be a good idea though, but you might want to give a hint to people about trying hybrid as well if they want to optimize performance.

There is already a log line about setting this, I think we could add more info there. Perhaps a link to the miopen docs. But maybe it's enough just mentioning the env var, users can search for it easily enough.

The crux of this PR is that cudnn is currently disabled because of the impact on initial runs. I don't have this issue myself on rocm 6.4 and disabling cudnn significantly reduces ImageUpscaleWithModel performance (#10447) for me. This PR is an attempt at a better workaround.

[comfyanonymous]

I tried this, it doesn't change anything on my setups.

[alexheretic]

Ok fair enough. Do you have a rocm 6.4 test case? Perhaps we can enable cudnn for that version?