Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] Refactor Attention Take 2 #3462

Merged
merged 88 commits into from
Mar 25, 2024
Merged

[Core] Refactor Attention Take 2 #3462

merged 88 commits into from
Mar 25, 2024

Commits on Mar 17, 2024

  1. Move

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    aa4b6c0 View commit details
    Browse the repository at this point in the history

  2. Attention

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    cf1c96c View commit details
    Browse the repository at this point in the history

  3. Fix

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    88f1ff6 View commit details
    Browse the repository at this point in the history

  4. Fix import errors

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    2044252 View commit details
    Browse the repository at this point in the history

  5. Remove

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    6378222 View commit details
    Browse the repository at this point in the history

  6. Add Abstract AttentionBackend

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    65b5988 View commit details
    Browse the repository at this point in the history

  7. FlashAttn -> FlashInfer

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    6605797 View commit details
    Browse the repository at this point in the history

  8. Remove KVCache Layout

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    b6f2cec View commit details
    Browse the repository at this point in the history

  9. Add attention dispatcher

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    e5e0f49 View commit details
    Browse the repository at this point in the history

  10. Fix Abstract attention backend

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    d6985fa View commit details
    Browse the repository at this point in the history

  11. Minor

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    c76e5c1 View commit details
    Browse the repository at this point in the history

  12. Fix models

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    0d5bc56 View commit details
    Browse the repository at this point in the history

  13. FlashInfer -> FlashAttention

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    5dde0d7 View commit details
    Browse the repository at this point in the history

  14. PagedAttentionImpl -> PagedAttention

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    6bed6da View commit details
    Browse the repository at this point in the history

  15. Minor

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    deb02da View commit details
    Browse the repository at this point in the history

  16. WIP

    @WoosukKwon
    WoosukKwon committed Mar 17, 2024
    Configuration menu
    Copy the full SHA
    2fd5b2c View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2024

  1. Minor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    8ce58cb View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    32894ce View commit details
    Browse the repository at this point in the history

  3. Fix Neuron

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    1344e73 View commit details
    Browse the repository at this point in the history

  4. Remove cache events

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    144f6cb View commit details
    Browse the repository at this point in the history

  5. Remove

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    47c59b4 View commit details
    Browse the repository at this point in the history

  6. KVCache -> torch.Tensor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    61d2f63 View commit details
    Browse the repository at this point in the history

  7. yapf

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    f820e1c View commit details
    Browse the repository at this point in the history

  8. Move

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    d2c0bf8 View commit details
    Browse the repository at this point in the history

  9. Get attn backend

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    0ec1786 View commit details
    Browse the repository at this point in the history

  10. Move

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    ec1904f View commit details
    Browse the repository at this point in the history

  11. Fix

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    86918c8 View commit details
    Browse the repository at this point in the history

  12. Fix

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    19820f9 View commit details
    Browse the repository at this point in the history

  13. yapf

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    f0f6a96 View commit details
    Browse the repository at this point in the history

  14. type

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    15a36b7 View commit details
    Browse the repository at this point in the history

  15. Remove InputMetadata

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    90d91cd View commit details
    Browse the repository at this point in the history

  16. Fix PagedAttention

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    d569f5c View commit details
    Browse the repository at this point in the history

  17. Fix

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    f88877f View commit details
    Browse the repository at this point in the history

  18. Minor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    fbed6b0 View commit details
    Browse the repository at this point in the history

  19. InputMetadata -> AttentionMetadata

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    7b20793 View commit details
    Browse the repository at this point in the history

  20. Comment

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    a6062d3 View commit details
    Browse the repository at this point in the history

  21. Fix swap and copy

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    faa1806 View commit details
    Browse the repository at this point in the history

  22. Minor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    2e14d70 View commit details
    Browse the repository at this point in the history

  23. Comment

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    82ee3d6 View commit details
    Browse the repository at this point in the history

  24. Minor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    d95176a View commit details
    Browse the repository at this point in the history

  25. Fix FlashAttention backend

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    1ffbe21 View commit details
    Browse the repository at this point in the history

  26. yapf

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    53e49c2 View commit details
    Browse the repository at this point in the history

  27. Minor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    b3b99fd View commit details
    Browse the repository at this point in the history

  28. Minor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    de7f764 View commit details
    Browse the repository at this point in the history

  29. yapfg

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    3ee77fd View commit details
    Browse the repository at this point in the history

  30. Minor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    bbcd032 View commit details
    Browse the repository at this point in the history

  31. Minor refactor

    @WoosukKwon
    WoosukKwon committed Mar 18, 2024
    Configuration menu
    Copy the full SHA
    83fbfcd View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2024

  1. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    8a89930 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 20, 2024
    Configuration menu
    Copy the full SHA
    0da3a5d View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2024

  1. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    6142ce9 View commit details
    Browse the repository at this point in the history

  2. Minor

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    c6ea553 View commit details
    Browse the repository at this point in the history

  3. Fix

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    1740d1f View commit details
    Browse the repository at this point in the history

  4. Fix

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    ea434dd View commit details
    Browse the repository at this point in the history

  5. Fix

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    1c6c06e View commit details
    Browse the repository at this point in the history

  6. yapf

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    3cee5ad View commit details
    Browse the repository at this point in the history

  7. Delete unused logger

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    a5ac88e View commit details
    Browse the repository at this point in the history

  8. Remove max_size=1

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    f87ee0d View commit details
    Browse the repository at this point in the history

  9. Use max_prompt_len instead of max_seq_len

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    8b1dc2e View commit details
    Browse the repository at this point in the history

  10. Add PagedAttentionMetadata

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    55010d9 View commit details
    Browse the repository at this point in the history

  11. ruff

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    c4ff6ee View commit details
    Browse the repository at this point in the history

  12. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    ab2faeb View commit details
    Browse the repository at this point in the history

  13. Fix

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    bbb1f11 View commit details
    Browse the repository at this point in the history

  14. Fix Jais

    @WoosukKwon
    WoosukKwon committed Mar 21, 2024
    Configuration menu
    Copy the full SHA
    927f788 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2024

  1. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 22, 2024
    Configuration menu
    Copy the full SHA
    cbefadb View commit details
    Browse the repository at this point in the history

Commits on Mar 24, 2024

  1. Rename

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    5d87e8f View commit details
    Browse the repository at this point in the history

  2. Fix comment

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    64734f8 View commit details
    Browse the repository at this point in the history

  3. Fix comment

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    1888a3e View commit details
    Browse the repository at this point in the history

  4. Minor

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    5591caa View commit details
    Browse the repository at this point in the history

  5. Merge allocate_gpu_cache and allocate_cpu_cache

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    fba63c0 View commit details
    Browse the repository at this point in the history

  6. Fix comment on ref attention

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    786110d View commit details
    Browse the repository at this point in the history

  7. Minor

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    4cbae94 View commit details
    Browse the repository at this point in the history

  8. ref -> naive

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    587f11e View commit details
    Browse the repository at this point in the history

  9. Fix

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    d6964f7 View commit details
    Browse the repository at this point in the history

  10. PagedAttention.get_kv_cache_shape

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    a49e87d View commit details
    Browse the repository at this point in the history

  11. Empty cache

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    0ce6259 View commit details
    Browse the repository at this point in the history

  12. use enforce_eager for test_logprob

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    843cb16 View commit details
    Browse the repository at this point in the history

  13. Revert

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    76552c4 View commit details
    Browse the repository at this point in the history

  14. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    0dbaed6 View commit details
    Browse the repository at this point in the history

  15. log when using naive attention

    @WoosukKwon
    WoosukKwon committed Mar 24, 2024
    Configuration menu
    Copy the full SHA
    6747d14 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2024

  1. Merge branch 'main' into flashinfer-take3

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    2f1db63 View commit details
    Browse the repository at this point in the history

  2. lru_cache(maxsize=None)

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    fffdeea View commit details
    Browse the repository at this point in the history

  3. Minor

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    f723ac1 View commit details
    Browse the repository at this point in the history

  4. Minor

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    722e09b View commit details
    Browse the repository at this point in the history

  5. Add assert

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    43bb346 View commit details
    Browse the repository at this point in the history

  6. Minor fix in err msg

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    12eba34 View commit details
    Browse the repository at this point in the history

  7. soft fail for samplers test

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    e84fa8a View commit details
    Browse the repository at this point in the history

  8. Add explicit gc to test_beam_search

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    58de636 View commit details
    Browse the repository at this point in the history

  9. Minor

    @WoosukKwon
    WoosukKwon committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    6a8b538 View commit details
    Browse the repository at this point in the history