Commits

Commits on Dec 23, 2024

make LlamaModel._update_causal_mask torch compilable (#35187 )

winglian
and
ArthurZucker
authored

Commits on Dec 13, 2024

don't use no_sync when deepspeed doesn't support it for certain zero stages (#35157 )
winglian
authored

Commits on Nov 18, 2024

make sure to disable gradients for integer tensor (#32943 )
winglian
authored

Commits on Nov 15, 2024

FSDP grad accum fix (#34645 )
winglian
authored

Commits on Oct 1, 2024

fix check for hidden size in text model for deepspeed zero3 auto entries (#33829 )
winglian
authored

Commits on Sep 13, 2024

add a callback hook right before the optimizer step (#33444 )
winglian
authored

Commits on Sep 9, 2024

schedulefree optimizers (#30079 )

winglian
and
tmm1
authored

Commits on Aug 29, 2024

pass module to Params4bit.from_prequantized to ensure quant_state (#32524 )
winglian
authored

Commits on Jul 30, 2024

fixes to properly shard FSDP across cpu and meta for cpu_efficient_loading for prequantized 4bit (#32276 )
winglian
authored

Commits on Jun 28, 2024

don't zero out the attention_mask when using sliding window with flash attention (#31670 )
winglian
authored

Commits on Apr 23, 2024

fix for itemsize => element_size() for torch backwards compat (#30133 )

authored

Commits on May 2, 2023

Fix check for backword_pos (#23075 )
winglian
authored

Commits on Apr 21, 2023

ddp fixes for training (#22874 )
winglian
authored