Paged Low Bit Optimizers

[msaroufim]

Right now our Optimizers are low bit so they save a bunch of memory but considering optimizers can also spike memory it's common to page them out to CPU RAM. There's a prototype of this here #425 and can combo this idea with low bit optimizers

A halfway solution would be making sure our low bit optimizers work well on CPU

[jcaip]

cc @gau-nernst are you still planning on working on this?

[msaroufim]

to be fair I don't think @gau-nernst ever signed up for this haha

[gau-nernst]

@jcaip I don't plan to work on this. Anyone can take if interested.

[jcaip]

@gau-nernst sounds good - just asking as we've just been doing some old issue triage today :)