StableAdamW: Missing divisor in RMS computation

[EricZimmermann]

Hello

It seems as though there is a minor error in computing RMS in Stable Adam:

RMS is computed as:

fastxtend/fastxtend/optimizer/stableadam.py

Line 53 in e6b2c39

rms = torch.norm(p.grad.data.div(root_sqr_avg.maximum(eps_t)), 2)

And is missing a normalization coefficient for the mean:
rms = torch.norm(p.grad.data.div(root_sqr_avg.maximum(eps_t)), 2) / math.sqrt(p.grad.data.div.numel())

as per

Additionally, another eps can be added in the final normalization.

[warner-benjamin]

I'm in the process of porting over the fastxtend optimizers to use the optimi implementations, which should resolve this.

[warner-benjamin]

@EricZimmermann #28 tracks the progress of porting over the fastxtend optimizers to use the optimi implementations