it run good

[werruww]

Your current environment

sucvllm.txt

🐛 Describe the bug

#/default/create_completion_v1_completions_post

{
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"prompt": "Who is Napoleon Bonaparte?",
"max_tokens": 128,
"temperature": 0.0
}

{
"id": "cmpl-4b3d25bf255944828dfcdac816c28cb1",
"object": "text_completion",
"created": 1750894412,
"model": "Qwen/Qwen2.5-1.5B-Instruct",
"choices": [
{
"index": 0,
"text": " Who was he and what did he do?\nNapoleon Bonaparte, born on August 15, 1769, in Ajaccio, Corsica, was a French military leader who rose to prominence during the French Revolution. He served as First Consul of France from 1799 to 1804 and Emperor of the French from 1804 until his abdication in 1814. After briefly returning to power in 1815, he was exiled to Saint Helena, where he died.\n\nKey aspects of Napoleon's life and accomplishments include:\n\n",
"logprobs": null,
"finish_reason": "length",
"stop_reason": null,
"prompt_logprobs": null
}
],
"usage": {
"prompt_tokens": 7,
"total_tokens": 135,
"completion_tokens": 128,
"prompt_tokens_details": null
}
}

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[werruww]

python examples/offline_inference/basic/generate.py --model Qwen/Qwen2.5-1.5B-Instruct

(py312) m@m:~/vllm_source$ python examples/offline_inference/basic/generate.py --model Qwen/Qwen2.5-1.5B-Instruct
INFO 06-25 17:15:13 [init.py:239] Automatically detected platform cpu.
INFO 06-25 17:15:24 [config.py:713] This model supports multiple tasks: {'classify', 'embed', 'score', 'generate', 'reward'}. Defaulting to 'generate'.
WARNING 06-25 17:15:24 [arg_utils.py:1684] device type=cpu is not supported by the V1 Engine. Falling back to V0.
INFO 06-25 17:15:24 [config.py:1794] Disabled the custom all-reduce kernel because it is not supported on current platform.
WARNING 06-25 17:15:24 [cpu.py:106] Environment variable VLLM_CPU_KVCACHE_SPACE (GiB) for CPU backend is not set, using 4 by default.
WARNING 06-25 17:15:24 [cpu.py:119] uni is not supported on CPU, fallback to mp distributed executor backend.
INFO 06-25 17:15:24 [llm_engine.py:243] Initializing a V0 LLM engine (v0.8.5.dev120+g471fe6563) with config: model='Qwen/Qwen2.5-1.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-1.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen2.5-1.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False,
INFO 06-25 17:15:27 [cpu.py:45] Using Torch SDPA backend.
INFO 06-25 17:15:29 [parallel_state.py:946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 06-25 17:15:30 [weight_utils.py:265] Using model weights format ['*.safetensors']
INFO 06-25 17:15:31 [weight_utils.py:315] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:13<00:00, 13.95s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:13<00:00, 13.95s/it]

INFO 06-25 17:15:45 [loader.py:458] Loading weights took 14.06 seconds
INFO 06-25 17:15:45 [executor_base.py:112] # cpu blocks: 9362, # CPU blocks: 0
INFO 06-25 17:15:45 [executor_base.py:117] Maximum concurrency for 32768 tokens per request: 4.57x
INFO 06-25 17:15:45 [llm_engine.py:449] init engine (profile, create kv cache, warmup model) took 0.57 seconds
WARNING 06-25 17:15:45 [config.py:1201] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with --generation-config vllm.
Processed prompts: 0%| | 0/4 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]WARNING 06-25 17:15:46 [cpu.py:170] Pin memory is not supported on CPU.
Processed prompts: 100%|██████████████████████████████████████████████████| 4/4 [00:03<00:00, 1.17it/s, est. speed input: 6.44 toks/s, output: 18.74 toks/s]

Prompt: 'Hello, my name is'
Generated text: ' Katarina and I am 25 years old. I live in Bel'

Prompt: 'The president of the United States is'
Generated text: ' a very powerful leader. The President makes decisions about many things, such as what'

Prompt: 'The capital of France is'
Generated text: ' a city that has been the site of many important historical events and political changes.'

Prompt: 'The future of AI is'
Generated text: ' not just about robots and drones, it’s also about the way we interact with'

(py312) m@m:~/vllm_source$

[werruww]

---

(py312) m@m:~/vllm_source$ python examples/offline_inference/basic/generate.py -h
INFO 06-25 17:27:13 [init.py:239] Automatically detected platform cpu.
usage: generate.py [-h] [--model MODEL] [--task {auto,generate,embedding,embed,classify,score,reward,transcription}] [--tokenizer TOKENIZER]
[--hf-config-path HF_CONFIG_PATH] [--skip-tokenizer-init] [--revision REVISION] [--code-revision CODE_REVISION]
[--tokenizer-revision TOKENIZER_REVISION] [--tokenizer-mode {auto,slow,mistral,custom}] [--trust-remote-code]
[--allowed-local-media-path ALLOWED_LOCAL_MEDIA_PATH]
[--load-format {auto,pt,safetensors,npcache,dummy,tensorizer,sharded_state,gguf,bitsandbytes,mistral,runai_streamer,fastsafetensors}]
[--download-dir DOWNLOAD_DIR] [--model-loader-extra-config MODEL_LOADER_EXTRA_CONFIG] [--use-tqdm-on-load | --no-use-tqdm-on-load]
[--config-format {auto,hf,mistral}] [--dtype {auto,half,float16,bfloat16,float,float32}] [--max-model-len MAX_MODEL_LEN]
[--guided-decoding-backend {auto,outlines,lm-format-enforcer,xgrammar}] [--reasoning-parser {deepseek_r1,granite}]
[--logits-processor-pattern LOGITS_PROCESSOR_PATTERN] [--model-impl {auto,vllm,transformers}]
[--distributed-executor-backend {ray,mp,uni,external_launcher}] [--pipeline-parallel-size PIPELINE_PARALLEL_SIZE]
[--tensor-parallel-size TENSOR_PARALLEL_SIZE] [--data-parallel-size DATA_PARALLEL_SIZE]
[--enable-expert-parallel | --no-enable-expert-parallel] [--max-parallel-loading-workers MAX_PARALLEL_LOADING_WORKERS]
[--ray-workers-use-nsight | --no-ray-workers-use-nsight] [--disable-custom-all-reduce | --no-disable-custom-all-reduce]
[--block-size {8,16,32,64,128}] [--gpu-memory-utilization GPU_MEMORY_UTILIZATION] [--swap-space SWAP_SPACE]
[--kv-cache-dtype {auto,fp8,fp8_e4m3,fp8_e5m2}] [--num-gpu-blocks-override NUM_GPU_BLOCKS_OVERRIDE]
[--enable-prefix-caching | --no-enable-prefix-caching] [--prefix-caching-hash-algo {builtin,sha256}] [--cpu-offload-gb CPU_OFFLOAD_GB]
[--calculate-kv-scales | --no-calculate-kv-scales] [--disable-sliding-window] [--use-v2-block-manager] [--seed SEED]
[--max-logprobs MAX_LOGPROBS] [--disable-log-stats]
[--quantization {aqlm,awq,deepspeedfp,tpu_int8,fp8,ptpc_fp8,fbgemm_fp8,modelopt,nvfp4,marlin,gguf,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,compressed-tensors,bitsandbytes,qqq,hqq,experts_int8,neuron_quant,ipex,quark,moe_wna16,torchao,None}]
[--rope-scaling ROPE_SCALING] [--rope-theta ROPE_THETA] [--hf-token [HF_TOKEN]] [--hf-overrides HF_OVERRIDES] [--enforce-eager]
[--max-seq-len-to-capture MAX_SEQ_LEN_TO_CAPTURE] [--tokenizer-pool-size TOKENIZER_POOL_SIZE] [--tokenizer-pool-type {ray}]
[--tokenizer-pool-extra-config TOKENIZER_POOL_EXTRA_CONFIG] [--limit-mm-per-prompt LIMIT_MM_PER_PROMPT]
[--mm-processor-kwargs MM_PROCESSOR_KWARGS] [--disable-mm-preprocessor-cache] [--enable-lora] [--enable-lora-bias]
[--max-loras MAX_LORAS] [--max-lora-rank MAX_LORA_RANK] [--lora-extra-vocab-size LORA_EXTRA_VOCAB_SIZE]
[--lora-dtype {auto,float16,bfloat16}] [--long-lora-scaling-factors LONG_LORA_SCALING_FACTORS] [--max-cpu-loras MAX_CPU_LORAS]
[--fully-sharded-loras] [--enable-prompt-adapter] [--max-prompt-adapters MAX_PROMPT_ADAPTERS]
[--max-prompt-adapter-token MAX_PROMPT_ADAPTER_TOKEN] [--device {auto,cuda,neuron,cpu,tpu,xpu,hpu}]
[--num-scheduler-steps NUM_SCHEDULER_STEPS] [--speculative-config SPECULATIVE_CONFIG] [--ignore-patterns IGNORE_PATTERNS]
[--preemption-mode PREEMPTION_MODE] [--served-model-name SERVED_MODEL_NAME [SERVED_MODEL_NAME ...]]
[--qlora-adapter-name-or-path QLORA_ADAPTER_NAME_OR_PATH] [--show-hidden-metrics-for-version SHOW_HIDDEN_METRICS_FOR_VERSION]
[--otlp-traces-endpoint OTLP_TRACES_ENDPOINT] [--collect-detailed-traces COLLECT_DETAILED_TRACES] [--disable-async-output-proc]
[--max-num-batched-tokens MAX_NUM_BATCHED_TOKENS] [--max-num-seqs MAX_NUM_SEQS] [--max-num-partial-prefills MAX_NUM_PARTIAL_PREFILLS]
[--max-long-partial-prefills MAX_LONG_PARTIAL_PREFILLS] [--long-prefill-token-threshold LONG_PREFILL_TOKEN_THRESHOLD]
[--num-lookahead-slots NUM_LOOKAHEAD_SLOTS] [--scheduler-delay-factor SCHEDULER_DELAY_FACTOR]
[--enable-chunked-prefill | --no-enable-chunked-prefill] [--multi-step-stream-outputs | --no-multi-step-stream-outputs]
[--scheduling-policy {fcfs,priority}] [--disable-chunked-mm-input | --no-disable-chunked-mm-input] [--scheduler-cls SCHEDULER_CLS]
[--override-neuron-config OVERRIDE_NEURON_CONFIG] [--override-pooler-config OVERRIDE_POOLER_CONFIG]
[--compilation-config COMPILATION_CONFIG] [--kv-transfer-config KV_TRANSFER_CONFIG] [--worker-cls WORKER_CLS]
[--worker-extension-cls WORKER_EXTENSION_CLS] [--generation-config GENERATION_CONFIG]
[--override-generation-config OVERRIDE_GENERATION_CONFIG] [--enable-sleep-mode] [--additional-config ADDITIONAL_CONFIG]
[--enable-reasoning] [--disable-cascade-attn] [--max-tokens MAX_TOKENS] [--temperature TEMPERATURE] [--top-p TOP_P] [--top-k TOP_K]

options:
-h, --help show this help message and exit

Engine arguments:
--additional-config ADDITIONAL_CONFIG
Additional config for specified platform in JSON format. Different platforms may support different configs. Make sure the configs
are valid for the platform you are using. The input format is like '{"config_key":"config_value"}' (default: None)
--allowed-local-media-path ALLOWED_LOCAL_MEDIA_PATH
Allowing API requests to read local images or videos from directories specified by the server file system. This is a security risk.
Should only be enabled in trusted environments. (default: None)
--code-revision CODE_REVISION
The specific revision to use for the model code on Hugging Face Hub. It can be a branch name, a tag name, or a commit id. If
unspecified, will use the default version. (default: None)
--collect-detailed-traces COLLECT_DETAILED_TRACES
Valid choices are model,worker,all. It makes sense to set this only if --otlp-traces-endpoint is set. If set, it will collect
detailed traces for the specified modules. This involves use of possibly costly and or blocking operations and hence might have a
performance impact. (default: None)
--compilation-config COMPILATION_CONFIG, -O COMPILATION_CONFIG
torch.compile configuration for the model.When it is a number (0, 1, 2, 3), it will be interpreted as the optimization level. NOTE:
level 0 is the default level without any optimization. level 1 and 2 are for internal testing only. level 3 is the recommended
level for production. To specify the full compilation config, use a JSON string, e.g. {"level": 3, "cudagraph_capture_sizes": [1, 2, 4, 8]} Following the convention of traditional compilers, using -O without space is also supported. -O3 is equivalent
to -O 3. (default: None)
--config-format {auto,hf,mistral}
The format of the model config to load.
* "auto" will try to load the config in hf format if available else it will try to load in mistral format (default:
ConfigFormat.AUTO)
--disable-async-output-proc
Disable async output processing. This may result in lower performance. (default: False)
--disable-cascade-attn
Disable cascade attention for V1. While cascade attention does not change the mathematical correctness, disabling it could be
useful for preventing potential numerical issues. Note that even if this is set to False, cascade attention will be only used when
the heuristic tells that it's beneficial. (default: False)
--disable-log-stats Disable logging statistics. (default: False)
--disable-mm-preprocessor-cache
If true, then disables caching of the multi-modal preprocessor/mapper. (not recommended) (default: False)
--disable-sliding-window
Disables sliding window, capping to sliding window size. (default: False)
--dtype {auto,half,float16,bfloat16,float,float32}
Data type for model weights and activations.
* "auto" will use FP16 precision for FP32 and FP16 models, and BF16 precision for BF16 models. * "half" for FP16. Recommended for
AWQ quantization. * "float16" is the same as "half". * "bfloat16" for a balance between precision and range. * "float" is shorthand
for FP32 precision. * "float32" for FP32 precision. (default: auto)
--enable-lora If True, enable handling of LoRA adapters. (default: False)
--enable-lora-bias If True, enable bias for LoRA adapters. (default: False)
--enable-prompt-adapter
If True, enable handling of PromptAdapters. (default: False)
--enable-reasoning Whether to enable reasoning_content for the model. If enabled, the model will be able to generate reasoning content. (default:
False)
--enable-sleep-mode Enable sleep mode for the engine. (only cuda platform is supported) (default: False)
--enforce-eager Always use eager-mode PyTorch. If False, will use eager mode and CUDA graph in hybrid for maximal performance and flexibility.
(default: False)
--fully-sharded-loras
By default, only half of the LoRA computation is sharded with tensor parallelism. Enabling this will use the fully sharded layers.
At high sequence length, max rank or tensor parallel size, this is likely faster. (default: False)
--generation-config GENERATION_CONFIG
The folder path to the generation config. Defaults to 'auto', the generation config will be loaded from model path. If set to
'vllm', no generation config is loaded, vLLM defaults will be used. If set to a folder path, the generation config will be loaded
from the specified folder path. If max_new_tokens is specified in generation config, then it sets a server-wide limit on the
number of output tokens for all requests. (default: auto)
--hf-config-path HF_CONFIG_PATH
Name or path of the huggingface config to use. If unspecified, model name or path will be used. (default: None)
--hf-overrides HF_OVERRIDES
Extra arguments for the HuggingFace config. This should be a JSON string that will be parsed into a dictionary. (default: None)
--hf-token [HF_TOKEN]
The token to use as HTTP bearer authorization for remote files. If True, will use the token generated when running huggingface- cli login (stored in ~/.huggingface). (default: None)
--ignore-patterns IGNORE_PATTERNS
The pattern(s) to ignore when loading the model.Default to original/**/* to avoid repeated loading of llama's checkpoints.
(default: [])
--kv-transfer-config KV_TRANSFER_CONFIG
The configurations for distributed KV cache transfer. Should be a JSON string. (default: None)
--logits-processor-pattern LOGITS_PROCESSOR_PATTERN
Optional regex pattern specifying valid logits processor qualified names that can be passed with the logits_processors extra
completion argument. Defaults to None, which allows no processors. (default: None)
--long-lora-scaling-factors LONG_LORA_SCALING_FACTORS
Specify multiple scaling factors (which can be different from base model scaling factor - see eg. Long LoRA) to allow for multiple
LoRA adapters trained with those scaling factors to be used at the same time. If not specified, only adapters trained with the base
model scaling factor are allowed. (default: None)
--lora-dtype {auto,float16,bfloat16}
Data type for LoRA. If auto, will default to base model dtype. (default: auto)
--lora-extra-vocab-size LORA_EXTRA_VOCAB_SIZE
Maximum size of extra vocabulary that can be present in a LoRA adapter (added to the base model vocabulary). (default: 256)
--max-cpu-loras MAX_CPU_LORAS
Maximum number of LoRAs to store in CPU memory. Must be >= than max_loras. (default: None)
--max-logprobs MAX_LOGPROBS
Max number of log probs to return logprobs is specified in SamplingParams. (default: 20)
--max-lora-rank MAX_LORA_RANK
Max LoRA rank. (default: 16)
--max-loras MAX_LORAS
Max number of LoRAs in a single batch. (default: 1)
--max-model-len MAX_MODEL_LEN
Model context length. If unspecified, will be automatically derived from the model config. Supports k/m/g/K/M/G in human-readable
format. Examples: - 1k → 1000 - 1K → 1024 (default: None)
--max-prompt-adapter-token MAX_PROMPT_ADAPTER_TOKEN
Max number of PromptAdapters tokens (default: 0)
--max-prompt-adapters MAX_PROMPT_ADAPTERS
Max number of PromptAdapters in a batch. (default: 1)
--max-seq-len-to-capture MAX_SEQ_LEN_TO_CAPTURE
Maximum sequence length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode.
Additionally for encoder-decoder models, if the sequence length of the encoder input is larger than this, we fall back to the eager
mode. (default: 8192)
--mm-processor-kwargs MM_PROCESSOR_KWARGS
Overrides for the multimodal input mapping/processing, e.g., image processor. For example: {"num_crops": 4}. (default: None)
--model MODEL Name or path of the huggingface model to use. (default: meta-llama/Llama-3.2-1B-Instruct)
--model-impl {auto,vllm,transformers}
Which implementation of the model to use.
* "auto" will try to use the vLLM implementation if it exists and fall back to the Transformers implementation if no vLLM
implementation is available. * "vllm" will use the vLLM model implementation. * "transformers" will use the Transformers model
implementation. (default: auto)
--num-scheduler-steps NUM_SCHEDULER_STEPS
Maximum number of forward steps per scheduler call. (default: 1)
--otlp-traces-endpoint OTLP_TRACES_ENDPOINT
Target URL to which OpenTelemetry traces will be sent. (default: None)
--override-generation-config OVERRIDE_GENERATION_CONFIG
Overrides or sets generation config in JSON format. e.g. {"temperature": 0.5}. If used with --generation-config=auto, the
override parameters will be merged with the default config from the model. If generation-config is None, only the override
parameters are used. (default: None)
--override-neuron-config OVERRIDE_NEURON_CONFIG
Override or set neuron device configuration. e.g. {"cast_logits_dtype": "bloat16"}. (default: None)
--override-pooler-config OVERRIDE_POOLER_CONFIG
Override or set the pooling method for pooling models. e.g. {"pooling_type": "mean", "normalize": false}. (default: None)
--preemption-mode PREEMPTION_MODE
If 'recompute', the engine performs preemption by recomputing; If 'swap', the engine performs preemption by block swapping.
(default: None)
--qlora-adapter-name-or-path QLORA_ADAPTER_NAME_OR_PATH
Name or path of the QLoRA adapter. (default: None)
--quantization {aqlm,awq,deepspeedfp,tpu_int8,fp8,ptpc_fp8,fbgemm_fp8,modelopt,nvfp4,marlin,gguf,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,compressed-tensors,bitsandbytes,qqq,hqq,experts_int8,neuron_quant,ipex,quark,moe_wna16,torchao,None}, -q {aqlm,awq,deepspeedfp,tpu_int8,fp8,ptpc_fp8,fbgemm_fp8,modelopt,nvfp4,marlin,gguf,gptq_marlin_24,gptq_marlin,awq_marlin,gptq,compressed-tensors,bitsandbytes,qqq,hqq,experts_int8,neuron_quant,ipex,quark,moe_wna16,torchao,None}
Method used to quantize the weights. If None, we first check the quantization_config attribute in the model config file. If that
is None, we assume the model weights are not quantized and use dtype to determine the data type of the weights. (default: None)
--revision REVISION The specific model version to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the default
version. (default: None)
--rope-scaling ROPE_SCALING
RoPE scaling configuration in JSON format. For example, {"rope_type":"dynamic","factor":2.0} (default: None)
--rope-theta ROPE_THETA
RoPE theta. Use with rope_scaling. In some cases, changing the RoPE theta improves the performance of the scaled model. (default:
None)
--scheduler-cls SCHEDULER_CLS
The scheduler class to use. "vllm.core.scheduler.Scheduler" is the default scheduler. Can be a class directly or the path to a
class of form "mod.custom_class". (default: vllm.core.scheduler.Scheduler)
--seed SEED Random seed for operations. (default: None)
--served-model-name SERVED_MODEL_NAME [SERVED_MODEL_NAME ...]
The model name(s) used in the API. If multiple names are provided, the server will respond to any of the provided names. The model
name in the model field of a response will be the first name in this list. If not specified, the model name will be the same as the
--model argument. Noted that this name(s) will also be used in model_name tag content of prometheus metrics, if multiple
names provided, metrics tag will take the first one. (default: None)
--show-hidden-metrics-for-version SHOW_HIDDEN_METRICS_FOR_VERSION
Enable deprecated Prometheus metrics that have been hidden since the specified version. For example, if a previously deprecated
metric has been hidden since the v0.7.0 release, you use --show-hidden-metrics-for-version=0.7 as a temporary escape hatch while
you migrate to new metrics. The metric is likely to be removed completely in an upcoming release. (default: None)
--skip-tokenizer-init
Skip initialization of tokenizer and detokenizer. Expects valid prompt_token_ids and None for prompt from the input. The generated
output will contain token ids. (default: False)
--speculative-config SPECULATIVE_CONFIG
The configurations for speculative decoding. Should be a JSON string. (default: None)
--task {auto,generate,embedding,embed,classify,score,reward,transcription}
The task to use the model for. Each vLLM instance only supports one task, even if the same model can be used for multiple tasks.
When the model only supports one task, "auto" can be used to select it; otherwise, you must specify explicitly which task to
use. (default: auto)
--tokenizer TOKENIZER
Name or path of the huggingface tokenizer to use. If unspecified, model name or path will be used. (default: None)
--tokenizer-mode {auto,slow,mistral,custom}
The tokenizer mode.
* "auto" will use the fast tokenizer if available. * "slow" will always use the slow tokenizer. * "mistral" will always use the
mistral_common tokenizer. * "custom" will use --tokenizer to select the preregistered tokenizer. (default: auto)
--tokenizer-revision TOKENIZER_REVISION
Revision of the huggingface tokenizer to use. It can be a branch name, a tag name, or a commit id. If unspecified, will use the
default version. (default: None)
--trust-remote-code Trust remote code from huggingface. (default: False)
--use-v2-block-manager
[DEPRECATED] block manager v1 has been removed and SelfAttnBlockSpaceManager (i.e. block manager v2) is now the default. Setting
this flag to True or False has no effect on vLLM behavior. (default: True)
--worker-cls WORKER_CLS
The worker class to use for distributed execution. (default: auto)
--worker-extension-cls WORKER_EXTENSION_CLS
The worker extension class on top of the worker cls, it is useful if you just want to add new functions to the worker class without
changing the existing functions. (default: )

Sampling parameters:
--max-tokens MAX_TOKENS
--temperature TEMPERATURE
--top-k TOP_K
--top-p TOP_P
(py312) m@m:~/vllm_source$

[werruww]

it run on cpu wsl

[werruww]

(py312) m@m:~/vllm_source$ python examples/offline_inference/basic/chat.py --model Qwen/Qwen2.5-1.5B-Instruct
INFO 06-25 17:38:01 [init.py:239] Automatically detected platform cpu.
INFO 06-25 17:38:13 [config.py:713] This model supports multiple tasks: {'embed', 'classify', 'generate', 'reward', 'score'}. Defaulting to 'generate'.
WARNING 06-25 17:38:13 [arg_utils.py:1684] device type=cpu is not supported by the V1 Engine. Falling back to V0.
INFO 06-25 17:38:13 [config.py:1794] Disabled the custom all-reduce kernel because it is not supported on current platform.
WARNING 06-25 17:38:13 [cpu.py:106] Environment variable VLLM_CPU_KVCACHE_SPACE (GiB) for CPU backend is not set, using 4 by default.
WARNING 06-25 17:38:13 [cpu.py:119] uni is not supported on CPU, fallback to mp distributed executor backend.
INFO 06-25 17:38:13 [llm_engine.py:243] Initializing a V0 LLM engine (v0.8.5.dev120+g471fe6563) with config: model='Qwen/Qwen2.5-1.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-1.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=Qwen/Qwen2.5-1.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False,
INFO 06-25 17:38:16 [cpu.py:45] Using Torch SDPA backend.
INFO 06-25 17:38:18 [parallel_state.py:946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 06-25 17:38:18 [weight_utils.py:265] Using model weights format ['*.safetensors']
INFO 06-25 17:38:19 [weight_utils.py:315] No model.safetensors.index.json found in remote.
Loading safetensors checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:13<00:00, 13.65s/it]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:13<00:00, 13.65s/it]

INFO 06-25 17:38:32 [loader.py:458] Loading weights took 13.73 seconds
INFO 06-25 17:38:32 [executor_base.py:112] # cpu blocks: 9362, # CPU blocks: 0
INFO 06-25 17:38:32 [executor_base.py:117] Maximum concurrency for 32768 tokens per request: 4.57x
INFO 06-25 17:38:33 [llm_engine.py:449] init engine (profile, create kv cache, warmup model) took 0.55 seconds
WARNING 06-25 17:38:33 [config.py:1201] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with --generation-config vllm.

INFO 06-25 17:38:35 [chat_utils.py:396] Detected the chat template content format to be 'string'. You can set --chat-template-content-format to override this.
WARNING 06-25 17:38:36 [cpu.py:170] Pin memory is not supported on CPU.
INFO 06-25 17:38:38 [metrics.py:489] Avg prompt throughput: 9.4 tokens/s, Avg generation throughput: 2.9 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

Generated Outputs:

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'The Importance of Higher Education\n\nHigher education has become increasingly important in modern society,'

Processed prompts: 0%| | 0/10 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]INFO 06-25 17:38:49 [metrics.py:489] Avg prompt throughput: 45.6 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 10 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.3%, CPU KV cache usage: 0.0%.
Processed prompts: 100%|███████████████████████████████████████████████| 10/10 [00:14<00:00, 1.41s/it, est. speed input: 34.03 toks/s, output: 11.34 toks/s]

Generated Outputs:

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'Higher education is an essential component of modern society, and its importance cannot be overst'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: "Higher education is crucial in today's world, as it plays a vital role in"

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'Higher education has become increasingly important in modern society as it equips individuals with knowledge'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'Certainly, here is my response:\n\nThe pursuit of higher education has long been recognized'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'The importance of higher education cannot be overstated in our modern world. With advancements'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'Higher education has become increasingly important in recent years, and for good reason. Higher'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'The Importance of Higher Education\n\nHigher education is often considered to be one of the'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'Certainly, here is my essay on the importance of higher education:\n\nHigher education has'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'Higher education is crucial for personal and professional growth, as it equips individuals with'

Prompt: '<|im_start|>system\nYou are a helpful assistant<|im_end|>\n<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\nHello! How can I assist you today?<|im_end|>\n<|im_start|>user\nWrite an essay about the importance of higher education.<|im_end|>\n<|im_start|>assistant\n'

Generated text: 'The importance of higher education cannot be overstated in our modern society. It is'

(py312) m@m:~/vllm_source$

[werruww]

(py312) m@m:~/vllm_source$ python examples/offline_inference/basic/generate.py --model facebook/opt-125m --dtype float32
INFO 06-25 17:49:24 [init.py:239] Automatically detected platform cpu.
INFO 06-25 17:49:27 [config.py:2943] Upcasting torch.float16 to torch.float32.
INFO 06-25 17:49:35 [config.py:713] This model supports multiple tasks: {'embed', 'classify', 'reward', 'generate', 'score'}. Defaulting to 'generate'.
WARNING 06-25 17:49:35 [arg_utils.py:1684] --dtype torch.float32 is not supported by the V1 Engine. Falling back to V0.
INFO 06-25 17:49:35 [config.py:1794] Disabled the custom all-reduce kernel because it is not supported on current platform.
WARNING 06-25 17:49:35 [cpu.py:106] Environment variable VLLM_CPU_KVCACHE_SPACE (GiB) for CPU backend is not set, using 4 by default.
WARNING 06-25 17:49:35 [cpu.py:119] uni is not supported on CPU, fallback to mp distributed executor backend.
INFO 06-25 17:49:35 [llm_engine.py:243] Initializing a V0 LLM engine (v0.8.5.dev120+g471fe6563) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float32, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=facebook/opt-125m, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=False, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":[],"compile_sizes":[],"cudagraph_capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=False,
INFO 06-25 17:49:37 [cpu.py:45] Using Torch SDPA backend.
INFO 06-25 17:49:39 [parallel_state.py:946] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 06-25 17:49:40 [weight_utils.py:265] Using model weights format ['*.bin']
Loading pt checkpoint shards: 0% Completed | 0/1 [00:00<?, ?it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 3.10it/s]
Loading pt checkpoint shards: 100% Completed | 1/1 [00:00<00:00, 3.09it/s]

INFO 06-25 17:49:40 [loader.py:458] Loading weights took 0.33 seconds
INFO 06-25 17:49:40 [executor_base.py:112] # cpu blocks: 3640, # CPU blocks: 0
INFO 06-25 17:49:40 [executor_base.py:117] Maximum concurrency for 2048 tokens per request: 28.44x
INFO 06-25 17:49:41 [llm_engine.py:449] init engine (profile, create kv cache, warmup model) took 0.51 seconds
Processed prompts: 0%| | 0/1 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]WARNING 06-25 17:49:41 [cpu.py:170] Pin memory is not supported on CPU.
Processed prompts: 100%|█████████████████████████████████████████████████| 1/1 [00:00<00:00, 2.43it/s, est. speed input: 19.44 toks/s, output: 38.87 toks/s]

Prompt: 'Who is Napoleon Bonaparte?'
Generated text: '\n\nBy Laura Jeanover\n\nMain content\n\nWe are glad you'

(py312) m@m:~/vllm_source$