Repository navigation
Wrong doc and function signature for 8-bit optim #1261
Description
System Info
NA
Reproduction
Argument optim_bits and amsgrad is not used for 8-bit Adam
The document for optim_bits is also wrong in this case, because we are using 8 bits here.
I'm thinking of 2 ways to solve this:
- Remove
optim_bitsandamsgradarguments (actually even 32-bit Adam also doesn't useamsgradargument) - If we want to keep the function signature the same, there should be a check that
optim_bits == 8andamsgrad == False
Expected behavior
NA
Activity
nighting0le01 commentedon Oct 24, 2024
? so does it always run in 8 bit?? @gau-nernst
gau-nernst commentedon Oct 24, 2024
Yes,
Adam8bitwill always run in 8 bit. Theoptim_bitsargument has no effectnighting0le01 commentedon Oct 24, 2024
@gau-nernst i see, thanks also, did you notice any difference varying min_8bit_size? would it make sense to start with such a high value if your hidden dim ==4096 and only goes above in projection layers?
gau-nernst commentedon Oct 24, 2024
I didn't play around with that parameter.
fix(optim): honor user-provided max_unorm in LAMB (+ 8-bit arg guards) (
kritikagarg commentedon Aug 28, 2026
Hi! I'd like to work on this issue.
Looking at the current state of the code,
Adam8bit(andAdamW8bit,PagedAdam8bit,PagedAdamW8bit) already have validation guards that reject non-default values foroptim_bitsandamsgrad. However, the docstrings are still misleading — they describe these as functional parameters with a small "Note:" appended, rather than clearly marking them as deprecated.I'm planning to:
.. deprecated::directives that clearly state these params exist only for signature compatibility and will be removedNo breaking changes — the params stay in the signature for backwards compatibility, just properly documented as deprecated.
Happy to adjust the approach based on feedback.