Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CPU] Add per-token asym INT8 dynamic quantization support to QKV/MLP node #27001

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open

[CPU] Add per-token asym INT8 dynamic quantization support to QKV/MLP node #27001

wants to merge 15 commits into from
+1,928 −403

Commits on Sep 30, 2024

  1. small batch opt

    @luo-cheng2021 @usstq
    luo-cheng2021 authored and usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    48ced7f View commit details
    Browse the repository at this point in the history

  2. fix that k is not a multiple of 256

    @luo-cheng2021 @usstq
    luo-cheng2021 authored and usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    39b8410 View commit details
    Browse the repository at this point in the history

  3. add qkv/mlp quantized mode

    @usstq
    usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    ecb1c31 View commit details
    Browse the repository at this point in the history

  4. fix quantize on nan float number

    @usstq
    usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    18665ab View commit details
    Browse the repository at this point in the history

  5. fix batch consistency bug in qkvproj

    @usstq
    usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a10b313 View commit details
    Browse the repository at this point in the history

  6. MLP&QKV support amx-f16

    @usstq
    usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    84a41fd View commit details
    Browse the repository at this point in the history

  7. remove linux_perf

    @usstq
    usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    a313502 View commit details
    Browse the repository at this point in the history

  8. add QKV/MLP dynamic quantization tests

    @usstq
    usstq committed Sep 30, 2024
    Configuration menu
    Copy the full SHA
    dbf235f View commit details
    Browse the repository at this point in the history

Commits on Oct 12, 2024

  1. fix coding style

    @usstq
    usstq committed Oct 12, 2024
    Configuration menu
    Copy the full SHA
    c5f9d5b View commit details
    Browse the repository at this point in the history

Commits on Oct 15, 2024

  1. Support LLM with qkv already fused like phi3

    @usstq
    usstq committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    16472f6 View commit details
    Browse the repository at this point in the history

  2. Support LLM with gate-up already fused like phi3

    @usstq
    usstq committed Oct 15, 2024
    Configuration menu
    Copy the full SHA
    36e1691 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Support dynamic-quant for combined qkv/mlp in LLM like phi3

    @usstq
    usstq committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    eaec512 View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2024

  1. Update

    @usstq
    usstq committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    b59ae1e View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2024

  1. using DYNAMIC_QUANTIZATION_GROUP_SIZE to enable per-token quant

    @usstq
    usstq committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    3c2dde9 View commit details
    Browse the repository at this point in the history

  2. fix CI error

    @usstq
    usstq committed Oct 18, 2024
    Configuration menu
    Copy the full SHA
    e09bf2d View commit details
    Browse the repository at this point in the history