Code

adding online benchmarking scripts#55

Code

Merged

tstescoTT merged 42 commits intomaintenstorrent/tt-inference-server:mainfrom tstesco/online-benchmarktenstorrent/tt-inference-server:tstesco/online-benchmarkCopy head branch name to clipboardDec 31, 2024

+1,437-518

Commits

Commits on Dec 20, 2024

• add print_prompts cli arg

  tstescoTT

  committed
  7d68656

  Copy full SHA for 7d68656

• remove redundant stop token from vLLM example api calls

  tstescoTT

  committed
  8d78d64

  Copy full SHA for 8d78d64

• add capture_trace.py util to pre-prompt vllm server to capture all trace input sizes

  tstescoTT

  committed
  3108bc0

  Copy full SHA for 3108bc0

• adding utils/startup_utils.py to refine handling of startup in automation

  tstescoTT

  committed
  ea3d75d

  Copy full SHA for ea3d75d

• adding force_max_tokens as option to call_inference_api(), add input_seq_lengths and output_seq_lengths directly args to test_api_call_threaded_full_queue() to allow for varied isl and osl within b…

  Show description for cc1d17a
  tstescoTT

  committed
  cc1d17a

  Copy full SHA for cc1d17a

• faster mock model prefill

  tstescoTT

  committed
  059d513

  Copy full SHA for 059d513

• make it not send stop tokens by default and speed up mock model decode and prefill

  tstescoTT

  committed
  48d17de

  Copy full SHA for 48d17de

• adding token count verification for vllm open ai api server to prompt_client_cli.py

  tstescoTT

  committed
  fead1aa

  Copy full SHA for fead1aa

• add max-log-len to limit logging of prompts to avoid clutter in logs

  tstescoTT

  committed
  5a80551

  Copy full SHA for 5a80551

• add InferenceServerContext to startup_utils.py, improve wait_for_healthy

  tstescoTT

  committed
  d845f08

  Copy full SHA for d845f08

• add all_responses to utils/prompt_client_cli.py not using globals

  tstescoTT

  committed
  632ac83

  Copy full SHA for 632ac83

• adding new utils/prompt_client_cli.py using utils/prompt_client.py and utils/batch_processor.py with configs in utils/prompt_configs.py and utils/prompt_generation.py for prompt generation

  tstescoTT

  committed
  f563e32

  Copy full SHA for f563e32

• fix health endpoint

  tstescoTT

  committed
  2467c74

  Copy full SHA for 2467c74

• add vllm_model to EnvironmentConfig instead of BatchConfig

  tstescoTT

  committed
  af5e8dc

  Copy full SHA for af5e8dc

• refactor utils/capture_traces.py with new prompt_client

  tstescoTT

  committed
  60c7ab2

  Copy full SHA for 60c7ab2

• fix utils imports

  tstescoTT

  committed
  10993a2

  Copy full SHA for 10993a2

• fix BatchConfig usage

  tstescoTT

  committed
  20ccdf4

  Copy full SHA for 20ccdf4

• add benchmarking/online_benchmark_prompt_client.py using prompt_client.py

  tstescoTT

  committed
  eab7e76

  Copy full SHA for eab7e76

• add benchmarking/online_benchmark_prompt_client.py using prompt_client.py

  tstescoTT

  committed
  90acdf6

  Copy full SHA for 90acdf6

• add benchmarking, evals, and tests dirs to Dockerfile

  tstescoTT

  committed
  ec486ad

  Copy full SHA for ec486ad

• update patchfile and benchmarking README.md with commands

  tstescoTT

  committed
  c58d7b3

  Copy full SHA for c58d7b3

• update Docker IMAGE_VERSION to v0.0.3

  tstescoTT

  committed
  fe4f96d

  Copy full SHA for fe4f96d

• improve doc

  tstescoTT

  committed
  f3d815a

  Copy full SHA for f3d815a

• update benchmark_serving.patch

  tstescoTT

  committed
  8246a72

  Copy full SHA for 8246a72

• add tt_model_runner.py patch for best_of

  tstescoTT

  committed
  765c4be

  Copy full SHA for 765c4be

• update benchmarking/benchmark_serving.patch

  tstescoTT

  committed
  b93370d

  Copy full SHA for b93370d

• use CACHE_ROOT for vllm_online_benchmark_results dir

  tstescoTT

  committed
  5e07baa

  Copy full SHA for 5e07baa

• adding timestamped online benchmark run result directory, rps=1 for vllm online benchmark script

  tstescoTT

  committed
  d0e0b0f

  Copy full SHA for d0e0b0f

• update benchmark output file naming convention

  tstescoTT

  committed
  5db2523

  Copy full SHA for 5db2523

• rename benchmarking/online_benchmark_prompt_client.py to benchmarking/prompt_client_online_benchmark.py

  tstescoTT

  committed
  5ab742c

  Copy full SHA for 5ab742c

• increase num_prompts default, default to 128/128 online test

  tstescoTT

  committed
  06420bd

  Copy full SHA for 06420bd

• use min_tokens and ignore_eos=True to force output seq len

  tstescoTT

  committed
  b7e4cfc

  Copy full SHA for b7e4cfc

• adding min_tokens to locust requests

  tstescoTT

  committed
  dda29a9

  Copy full SHA for dda29a9

• add --ignore-eos to vllm_online_benchmark.py to force the output seq len to be as configured

  tstescoTT

  committed
  f8b3033

  Copy full SHA for f8b3033

• add context_lens (isl, osl) pairs to capture_traces() to capture correct traces for performance testing

  tstescoTT

  committed
  12c38fc

  Copy full SHA for 12c38fc

• add trace pre-capture to prompt_client_cli.py with option to disable

  tstescoTT

  committed
  1cabdc9

  Copy full SHA for 1cabdc9

• better comment and logs for trace capture

  tstescoTT

  committed
  68f08d0

  Copy full SHA for 68f08d0

• use TPOT and TPS in benchmarking/prompt_client_online_benchmark.py, add support in client for ITL and TPOT

  tstescoTT

  committed
  962c507

  Copy full SHA for 962c507

• update utils/prompt_client_cli.py and docs

  tstescoTT

  committed
  62bf427

  Copy full SHA for 62bf427

• remove WIP utils/startup_utils.py from this branch

  tstescoTT

  committed
  d9e163c

  Copy full SHA for d9e163c

Commits on Dec 31, 2024

• adding doc string to BatchProcessor

  tstescoTT

  committed
  cd29085

  Copy full SHA for cd29085

• add output_path arg to batch_processor.py::BatchProcessor to optionally provide incremental output saveing for debugging, default to not saving output for benchmarking

  tstescoTT

  committed
  376403d

  Copy full SHA for 376403d