Raise an error if sync_dist is set when logging torchmetrics

[carmocca]

Proposal

Re-do #13903

---

Context

    > When using torchmetrics, it will sync whether you want it or not.

True, sorry, I noticed this later. We can make it None by default.

One can chose whether it happens additionally on step or not.

we can determine that based on on_step and on_epoch if sync_dist!=None:

• If either just on_step=True or just on_epoch=True, set the flag inside torchmetrics accordingly
• If on_step=True and on_epoch=True:
  • Set both the flags
  • Set only sync_on_compute to align it with how we handle scalar metrics.
  • Raise an error to ask user to set it by themselves, since we can't determine if they want to sync on_step, on_epoch or on both.

perhaps some of these checks are already done internally, but I wouldn't know where to look.

if you are talking about:

• in case torchmetrics are logged, what happens when sync_dist=True? It is ignored
• what happens on step level if sync_dist=True. Nothing.

Originally posted by @rohitgr7 in #13903 (comment)

cc @Borda @carmocca @Blaizzy

[shenoynikhil]

I'll take this up.

[justusschock]

@shenoynikhil how is it going here?

[shenoynikhil]

@justusschock Will attend to it this weekend.