[Bug] aisbench在构造请求体时是否会增加额外的内容，导致请求体大小比预估的大很多 #122
Description
操作系统及版本
ubuntu 22.04
安装工具的python环境
docker容器中的python环境
python版本
3.11
AISBench工具版本
3.0.20251103
AISBench执行命令
ais_bench --models vllm_api_stream_chat --datasets gsm8k_gen_0_shot_cot_str_perf --debug --summarizer default_perf --mode perf
模型配置文件或自定义配置文件内容
from ais_bench.benchmark.models import VLLMCustomAPIChatStream
from ais_bench.benchmark.utils.model_postprocessors import extract_non_reasoning_content
models = [
dict(
attr="service",
type=VLLMCustomAPIChatStream,
abbr='vllm-api-stream-chat',
path="/home/data/DeepSeek-V3.1_w8a8c8",
model="deepseek_r1_w8a8",
request_rate = 0.1,
retry = 2,
host_ip = "141.61.81.142",
host_port = 31015,
max_out_len = 1024,
batch_size=1,
trust_remote_code=False,
generation_kwargs = dict(
temperature = 0,
# top_k = 10,
# top_p = 0.95,
# seed = None,
# repetition_penalty = 1.03,
ignore_eos = True
),
pred_postprocessor=dict(type=extract_non_reasoning_content)
)
]
预期行为
请求成功发送到推理引擎MindIE，且不超过请求体1MB限制
实际行为
请求体过大，被推理引擎拦截，导致所有请求失败
01/23 16:23:35 - AISBench - INFO -
Request Per Second (RPS) Distribution Summary
Metric Value
Total Requests 16
Request Classification Normal: 16 | Timing Anomaly: 0 | Burstiness Anomaly: 0 | Infinite RPS Anomaly: 0
Target Rate 0.10 RPS
Burstiness 0.000
Normal RPS 0.10 ± 0.00
Normal RPS Range 0.10-0.10
Interval Stats Avg: 10.000s | Min: 10.000s | Max: 10.000s
Interval Classification Normal (Normal + Burstiness Anomaly): 16 | Anomaly (Timing Anomaly + Infinite RPS Anomaly): 0
[2026-01-23 16:23:35,283] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Calculate global interval offsets time: 0.2274 s
01/23 16:23:35 - AISBench - INFO - Process 0 using precomputed sleep offsets with 16 requests
Process-0 pid:18528: 0%| | 0/16 [00:00<?, ?it/s]01/23 16:23:45 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 6%|███████▌ 01/23 16:23:55 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 12%|███████████████▏ 01/23 16:24:05 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 19%|██████████████████████▋ 01/23 16:24:15 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 25%|██████████████████████████████▎ 01/23 16:24:25 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 31%|█████████████████████████████████████▊ 01/23 16:24:35 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 38%|█████████████████████████████████████████████▍ 01/23 16:24:45 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 44%|████████████████████████████████████████████████████▉ 01/23 16:24:55 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 50%|████████████████████████████████████████████████████████████▌ 01/23 16:25:05 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 56%|████████████████████████████████████████████████████████████████████ 01/23 16:25:15 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 62%|███████████████████████████████████████████████████████████████████████████ 01/23 16:25:25 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 69%|██████████████████████████████████████████████████████████████████████████████████▌ 01/23 16:25:35 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 75%|██████████████████████████████████████████████████████████████████████████████████████████ 01/23 16:25:45 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 81%|████████████████████████████████████████████████████████████████████████████████████████████01/23 16:25:55 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 88%|████████████████████████████████████████████████████████████████████████████████████████████01/23 16:26:05 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 94%|████████████████████████████████████████████████████████████████████████████████████████████01/23 16:26:15 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/clients/base_client.py - raise_error - 35 - [AisBenchClientException] Request failed: Exceeded maximum retry attempts (2).
Process-0 pid:18528: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [02:39<00:00, 9.98s/it]
Pid: 18528 | Post: 16 | Received: 0 | Failed: 16 | Post Time:160.02s | Receive Time:160.02s:
Encoding output text...: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 239674.51it/s]
01/23 16:26:15 - AISBench - INFO - Start converting origin data to detailed data ...
01/23 16:26:15 - AISBench - INFO - Finish converting origin data to detailed data | 0/16 [00:00<?, ?it/s]
01/23 16:26:15 - AISBench - INFO - Added 'Actual RPS: After Excluding Anomalies' to group 'Time - RPS: ' in legend explanation table
01/23 16:26:15 - AISBench - INFO - Successfully merged chart into position (1, 1)
01/23 16:26:15 - AISBench - INFO - RPS distribution charts saved to outputs/default/20260123_162329/performances/vllm-api-stream-chat/gsm8kdataset_rps_distribution_plot_with_actual_rps.html
01/23 16:26:15 - AISBench - INFO - Updated chart with actual RPS saved to outputs/default/20260123_162329/performances/vllm-api-stream-chat/gsm8kdataset_rps_distribution_plot_with_actual_rps.html
[2026-01-23 16:26:15,530] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Start extracting pref datas ...
[2026-01-23 16:26:15,531] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Finish extracting pref datas!
[2026-01-23 16:26:15,531] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Dumping detail perf data ...
Dumping data to h5: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 401.10it/s]
[2026-01-23 16:26:15,536] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Dump detail perf data cost: 0.005050430074334145(s)
[2026-01-23 16:26:15,536] [ais_bench.benchmark.openicl.icl_inferencer.icl_gen_perf_inferencer] [INFO] Performance task finished, results saved in outputs/default/20260123_162329/performances/vllm-api-stream-chat
01/23 16:26:15 - AISBench - INFO - time elapsed: 161.85s
Running tasks: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [02:47<00:00, 167.52s/it]
01/23 16:26:16 - AISBench - INFO - Performance evaluation tasks completed.
01/23 16:26:16 - AISBench - INFO - Loading detail perf data of model='vllm-api-stream-chat' dataset='gsm8kdataset' ...
01/23 16:26:16 - AISBench - INFO - Starting request timeline processing...
01/23 16:26:16 - AISBench - WARNING - No requests to plot!
01/23 16:26:16 - AISBench - ERROR - /usr/local/lib/python3.11/site-packages/ais_bench/benchmark/calculators/default_perf_metric_calculator.py - _init_datas - 21 - All requests failed, can't calculate performance results. Please check the ERROR log from every responses!
Traceback (most recent call last):
File "/usr/local/bin/ais_bench", line 8, in
sys.exit(main())
^^^^^^
File "/usr/local/lib/python3.11/site-packages/ais_bench/benchmark/cli/main.py", line 321, in main
summarizer = build_from_cfg(summarizer_cfg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ais_bench/benchmark/registry.py", line 87, in build_from_cfg
return PARTITIONERS.build(cfg)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/ais_bench/benchmark/summarizers/default_perf.py", line 69, in init
self._load_details_perf_data(calculator)
File "/usr/local/lib/python3.11/site-packages/ais_bench/benchmark/summarizers/default_perf.py", line 112, in _load_details_perf_data
calculators_per_model[dataset]._init_datas(details_data)
File "/usr/local/lib/python3.11/site-packages/ais_bench/benchmark/calculators/default_perf_metric_calculator.py", line 22, in _init_datas
raise ValueError("All requests failed!")
ValueError: All requests failed!
前置检查
- 我已读懂主页文档的快速入门，无法解决问题
- 我已检索过FAQ，无重复问题
- 我已搜索过现有Issue，无重复问题
- 我已更新到最新版本，问题仍存在