Monkeypatch for Phi3

[tyler-romero]

Summary

Add a new monkeypatch function to support patching Huggingface's Phi3 implementation with Liger Kernels.

Phi3 has its own MLP implementation (Phi3MLP) so a LigerPhi3SwiGLUMLP implementation that leverages LigerSiLUMulFunction is provided as well.

Testing Done

• run make test to ensure correctness
• run make checkstyle to ensure code style
• run make test-convergence to ensure convergence

Convergence test added (and passing on my 4090) for a minimodel based on Phi3 patched with liger kernels.
All tests passing.

Questions for Discussion

Apparently Phi3 was only added in transformers v4.41, but the lowest supported version of transformers in Liger-Kernel is 4.40.1. Additionally, only more recently has sdpa been supported in HF Phi3. Thoughts? Should I leave the transformers dependency version as-is?

[ByronHsu]

cc @shivam15s

[tyler-romero]

Sorry for the whitespace modifications here, this is the only change in this file

[yundai424]

thanks a lot for helping on the housekeeping! <3

[ByronHsu]

do you want to implement for FusedLinearCrossEntropy in the next PR and also create a issue to track? Essentially add a method patch

Liger-Kernel/src/liger_kernel/transformers/model/llama.py

Line 25 in 8aab06a

def lce_forward(

[tyler-romero]

Yeah happy to do that as well

[ByronHsu]

#98

[tyler-romero]

So Llama has its own MLP implementation here, but its named very generally. I went for a model-specific name here, but open to suggestions.

[ByronHsu]

yeah. this is necessary

[yundai424]

Thanks for this hyper speed PR woo hoo

[ByronHsu]

Apparently Phi3 was only added in transformers v4.41, but the lowest supported version of transformers in Liger-Kernel is 4.40.1. Additionally, only more recently has huggingface/transformers#32457. Thoughts? Should I leave the transformers dependency version as-is?

yeah let's bump to 4.41. sdpa seems not a hard requirement since most users use flash attn

[ByronHsu]

we can merge after transformers bump and fix conflict

[tyler-romero]

make test && make test-convergence
python -m pytest --disable-warnings test/ --ignore=test/convergence
=================================================================================================== test session starts ====================================================================================================
platform linux -- Python 3.10.13, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/tromero/workspace/Liger-Kernel
plugins: devtools-0.12.2
collected 138 items                                                                                                                                                                                                        

test/transformers/test_cross_entropy.py ........................................................ss                                                                                                                   [ 42%]
test/transformers/test_fused_linear_cross_entropy.py ......                                                                                                                                                          [ 46%]
test/transformers/test_geglu.py ........                                                                                                                                                                             [ 52%]
test/transformers/test_rms_norm.py ................................                                                                                                                                                  [ 75%]
test/transformers/test_rope.py ............                                                                                                                                                                          [ 84%]
test/transformers/test_swiglu.py ................                                                                                                                                                                    [ 95%]
test/transformers/test_trainer_integration.py ...                                                                                                                                                                    [ 97%]
test/transformers/test_transformers_monkey_patch.py .                                                                                                                                                                [ 98%]
test/triton/test_triton_monkey_patch.py ..                                                                                                                                                                           [100%]

======================================================================================== 136 passed, 2 skipped in 77.79s (0:01:17) =========================================================================================
HF_DATASETS_OFFLINE=1 python -m pytest --disable-warnings test/convergence
=================================================================================================== test session starts ====================================================================================================
platform linux -- Python 3.10.13, pytest-8.3.2, pluggy-1.5.0
rootdir: /home/tromero/workspace/Liger-Kernel
plugins: devtools-0.12.2
collected 12 items                                                                                                                                                                                                         

test/convergence/test_mini_models.py F.........                                                                                                                                                                      [ 83%]
test/convergence/test_mini_models_no_logits.py ..                                                                                                                                                                    [100%]

========================================================================================================= FAILURES =========================================================================================================
_____________________________________________________________________ test_mini_model[mini_gemma-32-0.0001-dtype0-1e-08-1e-05-0.005-1e-05-0.005-1e-05] _____________________________________________________________________

All relevant tests passing, mini_gemma test convergence is failing on my 4090 now though, perhaps related to the recent PR (#85):

>           raise AssertionError("\n".join(mismatch_details))
E           AssertionError: Number of mismatched elements: 3
E           Mismatch at index (0, 29): tensor1[(0, 29)] = 0.3853762447834015, tensor2[(0, 29)] = 0.38537222146987915
E           Mismatch at index (0, 30): tensor1[(0, 30)] = 0.7438472509384155, tensor2[(0, 30)] = 0.7438388466835022
E           Mismatch at index (0, 31): tensor1[(0, 31)] = 0.5316240191459656, tensor2[(0, 31)] = 0.5316107869148254

[tyler-romero]

I think I'll need a review from @yundai424 as well to bypass the "changes requested" flag

[lancerts]

@yundai424 can you take a look? ty

[ByronHsu]

lgtm and we can fix gemma in #92