Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Misc] Refactor benchmark_throughput.py #9779

Open
wants to merge 8 commits into
base: main
Choose a base branch
from
Open

[Misc] Refactor benchmark_throughput.py #9779

wants to merge 8 commits into from

Commits on Oct 29, 2024

  1. Refactor benchmark_throughput.py 

    * Give the request tuple a name
* Add helper message for --dataset flag

Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    Linkun Chen authored and Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    68cd83f View commit details
    Browse the repository at this point in the history

  2. Refactor benchmark_throughput.py to pass TextPrompt instead of string 

    * This is preparation to support multi-modality input, by reusing existing TextPrompt structure
* no significant metrics diff, see below
 - before: Throughput: 13.99 requests/s, 2933.11 total tokens/s, 2758.10 output tokens/s
 - after: Throughput: 13.99 requests/s, 2932.69 total tokens/s, 2757.70 output tokens/s
 - test command: `python benchmarks/benchmark_throughput.py --model mistral-community/pixtral-12b  --max-model-len=8192 --dataset ../sharegpt4v_instruct_gpt4-vision_cap100k.json`

Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    Linkun Chen authored and Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    9329d8d View commit details
    Browse the repository at this point in the history

  3. Update benchmark_throughput.py to support image input 

    Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    Linkun Chen authored and Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    2623fea View commit details
    Browse the repository at this point in the history

  4. dbg 

    Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    Linkun Chen authored and Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    2378563 View commit details
    Browse the repository at this point in the history

  5. fix lint 

    Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    @lk-chen
    lk-chen authored and Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    917ccb3 View commit details
    Browse the repository at this point in the history

  6. fix lint 

    Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    @lk-chen
    lk-chen authored and Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    6cb2fa7 View commit details
    Browse the repository at this point in the history

  7. fix sort 

    Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    a0199b5 View commit details
    Browse the repository at this point in the history

  8. Revert "Update benchmark_throughput.py to support image input" 

    This reverts commit 2623fea.

Signed-off-by: Linkun Chen <github+anyscale@lkchen.net>
    Linkun Chen committed Oct 29, 2024
    Configuration menu
    Copy the full SHA
    eb6e01b View commit details
    Browse the repository at this point in the history