[release] 0.9.1rc1 release checklist

[Yikun]

Prepare v0.9.1rc1

1. model support

• Download models 2025.06.20

  • A2 4 cards to Ascend02 @shen-shanshan
  • 310P 8 cards to 310P @leo-pony

• Altlas A2 series 2025.06.21

  • PR validation: @shen-shanshan
  • Run passed on with CANN 8.1rc1 @shen-shanshan
  • Docker image e2e test @shen-shanshan
  • Run accuracy: gsm8k @shen-shanshan
  • Doc Turtorial: Multi-NPU (XXXX 72B) @shen-shanshan

• Altlas 300I DUO series 2025.06.21

  • [Platform] Add initial experimental support for Altlas 300I series #1333
  • Dockerfile 310P and github action workflow @wangxiyuan [CI] Add 310p image build/push CI #1318
    • TAG(ubuntu): main-310p, v0.9.1rc1-310p
  • 310 support: [Platform] Add support for Ascend 310P #914
  • 310 moe support: supoort 310p moe #1327
  • Run passed with CANN 8.1rc1 on 300I DUO @leo-pony
  • Doc Turtorial: Multi-NPU (300I DUO) @leo-pony [Doc] Add Atlas 300I series tutorials #1341
  • (Before 0627) Add CI infra for 310p

2. PR to be merged

0.9.1-dev:

• [v0.9.1][DP] Tiny fix of dp and update example #1277
• [0.9.1][Bugfix] fix oom issue in mla and enable mla_pa for deepseek mla decode #1311
• [0.9.1]support deepseek w4a8 quantization #1320

main：

• release note
• [V1][eagle3] Support eagle3 proposer for v1 #1032
• [Platform] Add initial experimental support for Altlas 300I series #1333
  • [Platform] Add support for Ascend 310P #914
  • [CI] Add 310p image build/push CI #1318
  • supoort 310p moe #1327

DNM:

• [Bugfix] fix env variable in dbo #1284
• Disaggregate prefill for kv cache register style #950
• [DP] Add max_num_tokens_across_dp to AscendMetadata to fix dp and update example #1273

[Yikun]

Test test on main:

Download image

310p

# First pull mirror to speedup
docker pull quay.nju.edu.cn/ascend/vllm-ascend:v0.9.1rc1-310p
docker pull quay.io/ascend/vllm-ascend:v0.9.1rc1-310p

A2

# First pull mirror to speedup
docker pull quay.nju.edu.cn/ascend/vllm-ascend:v0.9.1rc1
docker pull quay.io/ascend/vllm-ascend:v0.9.1rc1

Run with A2:

export IMAGE=quay.io/ascend/vllm-ascend:v0.9.1rc1
docker run --rm \
        --name vllm-ascend \
        --device /dev/davinci4 \
        --device /dev/davinci5 \
        --device /dev/davinci6 \
        --device /dev/davinci7 \
        --device /dev/davinci_manager \
        --device /dev/devmm_svm \
        --device /dev/hisi_hdc \
        -v /usr/local/dcmi:/usr/local/dcmi \
        -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
        -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
        -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
        -v /etc/ascend_install.info:/etc/ascend_install.info \
        -v /root/.cache:/root/.cache \
        -p 8000:8000 \
        -it $IMAGE bash

# Make sure the commits are included:
cd /vllm-workspace/vllm-ascend/;git log -1;cd -

Run with 310p:

export IMAGE=quay.io/ascend/vllm-ascend:main-310p
docker run --rm \
        --name vllm-ascend \
        --device /dev/davinci0 \
        --device /dev/davinci1 \
        --device /dev/davinci2 \
        --device /dev/davinci3 \
        --device /dev/davinci4 \
        --device /dev/davinci5 \
        --device /dev/davinci6 \
        --device /dev/davinci7 \
        --device /dev/davinci_manager \
        --device /dev/devmm_svm \
        --device /dev/hisi_hdc \
        -v /usr/local/dcmi:/usr/local/dcmi \
        -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
        -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
        -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
        -v /etc/ascend_install.info:/etc/ascend_install.info \
        -v /root/.cache:/root/.cache \
        -p 8000:8000 \
        -it $IMAGE bash

# Make sure the commits are included:
cd /vllm-workspace/vllm-ascend/;git log -1;cd -

API Test

export TP=4
VLLM_USE_V1=1 vllm serve /root/.cache/pangu-pro-moe-model \
    --tensor-parallel-size $TP \
    --swap-space 16 \
    --disable-log-stats  --disable-log-requests  \
    --trust-remote-code  --enforce-eager

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/root/.cache/pangu-pro-moe-model",
        "prompt": "The future of AI is",
        "max_tokens": 128,
        "temperature": 0
    }'

Accuracy test

export TP=4
VLLM_USE_V1=1 vllm serve /root/.cache/pangu-pro-moe-model \
    --tensor-parallel-size $TP \
    --swap-space 16 \
    --disable-log-stats  --disable-log-requests  \
    --trust-remote-code  --enforce-eager

# Remove limit when real test
lm_eval \
  --model local-completions \
  --model_args model=/root/.cache/pangu-pro-moe-model,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,trust_remote_code=True \
  --tasks gsm8k --batch_size auto --limit 10 \
  --output_path ./

Benchmark test

export TP=4
VLLM_USE_V1=1 vllm serve /root/.cache/pangu-pro-moe-model \
    --tensor-parallel-size $TP \
    --swap-space 16 \
    --disable-log-stats  --disable-log-requests  \
    --load-format dummy --trust-remote-code  --enforce-eager

# Benchmark
export MODEL=/root/.cache/pangu-pro-moe-model
export VLLM_USE_MODELSCOPE=true
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install -r /vllm-workspace/vllm-ascend/benchmarks/requirements-bench.txt
python3 /vllm-workspace/vllm/benchmarks/benchmark_serving.py --model $MODEL --dataset-name random \
         --random-input-len 200 --num-prompts 200 --request-rate 1 --trust_remote_code \
         --save-result --result-dir ./

[shen-shanshan]

Functional Test Result with A2

{"id":"cmpl-013558085d774d66bf30c704decb762a","object":"text_completion","created":1750472788,"model":"/root/.cache/pangu-pro-moe-model","choices":[{"index":0,"text":" not just about creating smarter machines but about fostering collaboration between humans and AI systems. This partnership can lead to more efficient problem-solving, innovative solutions, and a better quality of life for people around the globe.\n\nHowever, achieving this future requires addressing several challenges. Ethical considerations, such as bias in AI algorithms and privacy concerns, must be prioritized. Additionally, ensuring that AI technologies are accessible to all and do not exacerbate existing inequalities is crucial.\n\nIn conclusion, AI stands at the forefront of technological advancement, with vast potential to transform industries and everyday life. By embracing its opportunities while responsibly managing its risks, we can harn","logprobs":null,"finish_reason":"length","stop_reason":null,"prompt_logprobs":null}],"usage":{"prompt_tokens":6,"total_tokens":134,"completion_tokens":128,"prompt_tokens_details":null},"kv_transfer_params":null}

Performance Test Result with A2

============ Serving Benchmark Result ============
Successful requests:                     200       
Benchmark duration (s):                  218.59    
Total input tokens:                      39800     
Total generated tokens:                  25600     
Request throughput (req/s):              0.91      
Output token throughput (tok/s):         117.12    
Total Token throughput (tok/s):          299.19    
---------------Time to First Token----------------
Mean TTFT (ms):                          282.88    
Median TTFT (ms):                        269.58    
P99 TTFT (ms):                           499.65    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          119.91    
Median TPOT (ms):                        120.33    
P99 TPOT (ms):                           123.53    
---------------Inter-token Latency----------------
Mean ITL (ms):                           120.09    
Median ITL (ms):                         119.20    
P99 ITL (ms):                            249.35    
==================================================

[leo-pony]

310P（300I Duo）, qwen2.5-0.5,单卡，测试结果：27.33 toks/s，其中：

测试脚本：

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]

# Create a sampling params object.
sampling_params = SamplingParams(max_tokens=100, temperature=0.0)
# Create an LLM.
llm = LLM(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    max_model_len=4096,
    max_num_seqs=4,
    trust_remote_code=True,
    tensor_parallel_size=1,
    dtype="float16", # IMPORTANT cause some ATB ops cannot support bf16 on 310P
    disable_custom_all_reduce=True, # IMPORTANT cause 310p needed
    compilation_config={"custom_ops":["+rms_norm", "+rotary_embedding"]}, # IMPORTANT cause 310p needed custom ops
)

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

测试结果：

INFO 06-21 03:26:38 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 06-21 03:26:38 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via `pip install triton` to enable kernel compilation.
INFO 06-21 03:26:40 [__init__.py:39] Available plugins for group vllm.platform_plugins:
INFO 06-21 03:26:40 [__init__.py:41] - ascend -> vllm_ascend:register
INFO 06-21 03:26:40 [__init__.py:44] All plugins in this group will be loaded. Set `VLLM_PLUGINS` to control which plugins to load.
INFO 06-21 03:26:40 [__init__.py:235] Platform plugin ascend is activated
WARNING 06-21 03:26:44 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
WARNING 06-21 03:26:48 [registry.py:401] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 06-21 03:26:48 [registry.py:401] Model architecture Qwen2VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_vl:AscendQwen2VLForConditionalGeneration.
WARNING 06-21 03:26:48 [registry.py:401] Model architecture Qwen2_5_VLForConditionalGeneration is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen2_5_vl:AscendQwen2_5_VLForConditionalGeneration.
WARNING 06-21 03:26:48 [registry.py:401] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 06-21 03:26:48 [registry.py:401] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM.
WARNING 06-21 03:26:48 [registry.py:401] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-0.5B-Instruct
2025-06-21 03:26:50,110 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-0.5B-Instruct
2025-06-21 03:26:51,167 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 06-21 03:27:10 [config.py:823] This model supports multiple tasks: {'score', 'classify', 'generate', 'reward', 'embed'}. Defaulting to 'generate'.
WARNING 06-21 03:27:10 [config.py:3271] Casting torch.bfloat16 to torch.float16.
INFO 06-21 03:27:20 [arg_utils.py:1653] npu is experimental on VLLM_USE_V1=1. Falling back to V0 Engine.
INFO 06-21 03:27:20 [config.py:1980] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-21 03:27:20 [platform.py:164] Compilation disabled, using eager mode by default
INFO 06-21 03:27:20 [llm_engine.py:230] Initializing a V0 LLM engine (v0.9.1) with config: model='Qwen/Qwen2.5-0.5B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen2.5-0.5B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=4096, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen2.5-0.5B-Instruct, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=None, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["none","+rms_norm","+rotary_embedding"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":4,"local_cache_dir":null}, use_cached_outputs=False,
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-0.5B-Instruct
2025-06-21 03:27:21,040 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-0.5B-Instruct
2025-06-21 03:27:22,659 - modelscope - INFO - Target directory already exists, skipping creation.
WARNING 06-21 03:27:23 [utils.py:2737] Methods add_prompt_adapter,cache_config,compilation_config,current_platform,list_prompt_adapters,load_config,pin_prompt_adapter,remove_prompt_adapter not implemented in <vllm_ascend.worker.worker.NPUWorker object at 0xfffd5e109960>
INFO 06-21 03:27:28 [parallel_state.py:1065] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 06-21 03:27:28 [model_runner.py:995] Starting to load model Qwen/Qwen2.5-0.5B-Instruct...
.Downloading Model from https://www.modelscope.cn to directory: /root/.cache/modelscope/hub/models/Qwen/Qwen2.5-0.5B-Instruct
2025-06-21 03:27:40,832 - modelscope - INFO - Target directory already exists, skipping creation.
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.86it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.85it/s]

INFO 06-21 03:27:41 [default_loader.py:272] Loading weights took 0.62 seconds
INFO 06-21 03:27:42 [model_runner.py:1000] Loading model weights took 0.9278 GB
[rank0]:[W621 03:27:45.357611322 compiler_depend.ts:79] Warning: [Check][offset] Check input storage_offset[%ld] = 0 failed, result is untrustworthy151935 (function operator())
.INFO 06-21 03:27:54 [executor_base.py:113] # npu blocks: 25437, # CPU blocks: 2730
INFO 06-21 03:27:54 [executor_base.py:118] Maximum concurrency for 4096 tokens per request: 794.91x
INFO 06-21 03:27:54 [llm_engine.py:428] init engine (profile, create kv cache, warmup model) took 12.50 seconds
Adding requests: 100%|█████| 4/4 [00:00<00:00, 485.70it/s]
Processed prompts:   0%|                                                                                                                                                                                                                                                                        | 0/4 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|█████| 4/4 [00:14<00:00,  3.66s/it, est. speed input: 1.50 toks/s, output: 27.33 toks/s]
Prompt: 'Hello, my name is', Generated text: ' Alex and I am a 17 year old male. I have been diagnosed with a rare genetic disorder called X-linked recessive. I have been told that I will not be able to have children. I have been told that I will not be able to have children because of the disorder. I have been told that I will not be able to have children because of the disorder. I have been told that I will not be able to have children because of the disorder. I have been told that'
Prompt: 'The president of the United States is', Generated text: ' a very important person. He is the leader of the country. He is the boss of the country. He is the leader of the country. He is the boss of the country. He is the leader of the country. He is the boss of the country. He is the leader of the country. He is the boss of the country. He is the leader of the country. He is the boss of the country. He is the leader of the country. He is the boss of the country'
Prompt: 'The capital of France is', Generated text: ' Paris. It is the largest city in Europe and the third largest city in the world. It is located in the south of France, on the banks of the Seine River. It is situated on the Île de la Cité, which is a small island in the center of the city. The city is surrounded by the Seine River and the Mediterranean Sea. It is also surrounded by the Pyrenees mountains. The city is known for its beautiful architecture, its rich history, and its'
Prompt: 'The future of AI is', Generated text: ' in the hands of the people. The future of AI is in the hands of the people. The future of AI is in the hands of the people. The future of AI is in the hands of the people. The future of AI is in the hands of the people. The future of AI is in the hands of the people. The future of AI is in the hands of the people. The future of AI is in the hands of the people. The future of AI is in the hands of'

[leo-pony]

310p(300I Duo), 8卡 测试脚本：

``` import os from vllm import LLM, SamplingParams

os.environ["VLLM_USE_V1"] = "1"
os.environ["VLLM_WORKER_MULTIPROC_METHOD"] = "spawn"

if name == "main":

prompts = [
    "Hello, my name is",
    "The future of AI is",
]
sampling_params = SamplingParams(min_tokens=8, max_tokens=8, temperature=0.0)
llm = LLM(model="/home/mnj/pangu_model/pangu-pro-moe-model",
        tensor_parallel_size=8,
        max_num_batched_tokens=2048,
        gpu_memory_utilization=0.5,
        max_num_seqs=4,
        enforce_eager=True,
        trust_remote_code=True,
        max_model_len=1024,
        disable_custom_all_reduce=True,
        enable_expert_parallel=True,

        dtype="float16", # IMPORTANT cause some ATB ops cannot support bf16 on 310P
        compilation_config={"custom_ops":["+rms_norm", "+rotary_embedding"]}, # IMPORTANT cause 310p needed custom ops

        additional_config = {
            'ascend_scheduler_config': {
                'enabled': True,
                'enable_chunked_prefill' : False,
                'chunked_prefill_enabled': False
            }

        }
)

# Generate texts from the prompts.
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
    prompt = output.prompt
    generated_text = output.outputs[0].text
    print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

测试结果：

INFO 06-21 04:49:56 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 06-21 04:49:56 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via pip install triton to enable kernel compilation.
INFO 06-21 04:49:57 [init.py:39] Available plugins for group vllm.platform_plugins:
INFO 06-21 04:49:57 [init.py:41] - ascend -> vllm_ascend:register
INFO 06-21 04:49:57 [init.py:44] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 06-21 04:49:57 [init.py:235] Platform plugin ascend is activated
WARNING 06-21 04:50:01 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
WARNING 06-21 04:50:05 [registry.py:401] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 06-21 04:50:05 [registry.py:401] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 06-21 04:50:05 [registry.py:401] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM.
WARNING 06-21 04:50:05 [registry.py:401] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
INFO 06-21 04:50:23 [config.py:823] This model supports multiple tasks: {'classify', 'embed', 'score', 'reward', 'generate'}. Defaulting to 'generate'.
WARNING 06-21 04:50:23 [config.py:3271] Casting torch.bfloat16 to torch.float16.
WARNING 06-21 04:50:23 [arg_utils.py:1647] Detected VLLM_USE_V1=1 with npu. Usage should be considered experimental. Please report any issues on Github.
INFO 06-21 04:50:23 [config.py:1946] Defaulting to use mp for distributed inference
INFO 06-21 04:50:23 [config.py:1980] Disabled the custom all-reduce kernel because it is not supported on current platform.
INFO 06-21 04:50:23 [config.py:2195] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 06-21 04:50:23 [platform.py:164] Compilation disabled, using eager mode by default
WARNING 06-21 04:50:25 [tokenizer.py:262] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
WARNING 06-21 04:50:31 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: NVIDIA/nccl#1234
INFO 06-21 04:50:32 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 06-21 04:50:32 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via pip install triton to enable kernel compilation.
INFO 06-21 04:50:33 [init.py:39] Available plugins for group vllm.platform_plugins:
INFO 06-21 04:50:33 [init.py:41] - ascend -> vllm_ascend:register
INFO 06-21 04:50:33 [init.py:44] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 06-21 04:50:33 [init.py:235] Platform plugin ascend is activated
WARNING 06-21 04:50:37 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
INFO 06-21 04:50:40 [core.py:455] Waiting for init message from front-end.
INFO 06-21 04:50:40 [platform.py:164] Compilation disabled, using eager mode by default
WARNING 06-21 04:50:42 [registry.py:401] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.
WARNING 06-21 04:50:42 [registry.py:401] Model architecture DeepseekV2ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV2ForCausalLM.
WARNING 06-21 04:50:42 [registry.py:401] Model architecture DeepseekV3ForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_v2:CustomDeepseekV3ForCausalLM.
WARNING 06-21 04:50:42 [registry.py:401] Model architecture Qwen3MoeForCausalLM is already registered, and will be overwritten by the new model class vllm_ascend.models.qwen3_moe:CustomQwen3MoeForCausalLM.
INFO 06-21 04:50:42 [core.py:70] Initializing a V1 LLM engine (v0.9.1) with config: model='/home/mnj/pangu_model/pangu-pro-moe-model', speculative_config=None, tokenizer='/home/mnj/pangu_model/pangu-pro-moe-model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.float16, max_seq_len=1024, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, device_config=npu, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/home/mnj/pangu_model/pangu-pro-moe-model, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=False, use_async_output_proc=True, pooler_config=None, compilation_config={"level":0,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":["+rms_norm","+rotary_embedding"],"splitting_ops":[],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":0,"cudagraph_capture_sizes":[],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":0,"local_cache_dir":null}
WARNING 06-21 04:50:42 [multiproc_worker_utils.py:307] Reducing Torch parallelism from 96 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 06-21 04:50:42 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3, 4, 5, 6, 7], buffer_handle=(8, 16777216, 10, 'psm_47dea0c6'), local_subscribe_addr='ipc:///tmp/5c3c4701-eb3d-4f38-a766-c8ae682aa6eb', remote_subscribe_addr=None, remote_addr_ipv6=False)
WARNING 06-21 04:50:47 [env_override.py:17] NCCL_CUMEM_ENABLE is set to 0, skipping override. This may increase memory overhead with cudagraph+allreduce: NVIDIA/nccl#1234
INFO 06-21 04:50:48 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 06-21 04:50:48 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via pip install triton to enable kernel compilation.
INFO 06-21 04:50:48 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
INFO 06-21 04:50:48 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 06-21 04:50:48 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via pip install triton to enable kernel compilation.
INFO 06-21 04:50:50 [init.py:39] Available plugins for group vllm.platform_plugins:
INFO 06-21 04:50:50 [init.py:41] - ascend -> vllm_ascend:register
INFO 06-21 04:50:50 [init.py:44] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 06-21 04:50:50 [init.py:235] Platform plugin ascend is activated
WARNING 06-21 04:50:54 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
WARNING 06-21 04:50:58 [registry.py:401] Model architecture DeepSeekMTPModel is already registered, and will be overwritten by the new model class vllm_ascend.models.deepseek_mtp:CustomDeepSeekMTP.g override. This may increase memory overhead with cudagraph+allreduce: NVIDIA/nccl#1234
INFO 06-21 04:51:09 [importing.py:17] Triton not installed or not compatible; certain GPU-related functions will not be available.
WARNING 06-21 04:51:09 [importing.py:29] Triton is not installed. Using dummy decorators. Install it via pip install triton to enable kernel compilation.
INFO 06-21 04:51:10 [init.py:39] Available plugins for group vllm.platform_plugins:
INFO 06-21 04:51:10 [init.py:41] - ascend -> vllm_ascend:register
INFO 06-21 04:51:10 [init.py:44] All plugins in this group will be loaded. Set VLLM_PLUGINS to control which plugins to load.
INFO 06-21 04:51:10 [init.py:235] Platform plugin ascend is activated
INFO 06-21 04:51:11 [init.py:39] Available plugins for group vllm.platform_plugins:
WARNING 06-21 04:51:14 [_custom_ops.py:22] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
(VllmWorker rank=0 pid=238293) INFO 06-21 04:51:24 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer_handle=(7, 4194304, 6, 'psm_ed3420a8'), local_subscribe_addr='ipc:///tmp/7081d464-aac6-4a33-8ad3-fb30bfd9f2d6', remote_subscribe_addr=None, remote_addr_ipv6=False)
(VllmWorker rank=0 pid=238293) INFO 06-21 04:51:25 [parallel_state.py:1065] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorker rank=2 pid=238295) INFO 06-21 04:51:35 [model_runner_v1.py:1829] Starting to load model /home/mnj/pangu_model/pangu-pro-moe-model...
Loading safetensors checkpoint shards: 0% Completed | 0/29 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 3% Completed | 1/29 [00:01<00:39, 1.41s/it]
Loading safetensors checkpoint shards: 7% Completed | 2/29 [00:02<00:35, 1.31s/it]
Loading safetensors checkpoint shards: 10% Completed | 3/29 [00:04<00:34, 1.34s/it]
Loading safetensors checkpoint shards: 14% Completed | 4/29 [00:05<00:36, 1.45s/it]
Loading safetensors checkpoint shards: 90% Completed | 26/29 [00:28<00:03, 1.06s/it]
Loading safetensors checkpoint shards: 93% Completed | 27/29 [00:29<00:02, 1.12s/it]
Loading safetensors checkpoint shards: 97% Completed | 28/29 [00:30<00:00, 1.01it/s]
(VllmWorker rank=3 pid=238296) INFO 06-21 04:52:11 [default_loader.py:272] Loading weights took 31.28 seconds
(VllmWorker rank=5 pid=238298) INFO 06-21 04:52:11 [default_loader.py:272] Loading weights took 32.39 seconds
Loading safetensors checkpoint shards: 100% Completed | 29/29 [00:31<00:00, 1.08s/it]
Loading safetensors checkpoint shards: 100% Completed | 29/29 [00:31<00:00, 1.10s/it]
(VllmWorker rank=3 pid=238296) INFO 06-21 04:52:12 [model_runner_v1.py:1848] Loading model weights took 16.8952 GB
[rank2]:[W621 04:52:15.493742576 compiler_depend.ts:82] Warning: The oprator of add is executed, Currently High Accuracy but Low Performance OP with 64-bit has been used, Please Do Some Cast at Python Functions with 32-bit for Better Performance! (function operator())
.......[rank7]:[W621 04:52:16.889457554 compiler_depend.ts:82] Warning: The oprator of add is executed, Currently High Accuracy but Low Performance OP with 64-bit has been used, Please Do Some Cast at Python Functions with 32-bit for Better Performance! (function operator())
.................INFO 06-21 04:52:40 [kv_cache_utils.py:715] GPU KV cache size: 83,840 tokens
INFO 06-21 04:52:40 [kv_cache_utils.py:719] Maximum concurrency for 1,024 tokens per request: 91.75x
INFO 06-21 04:52:40 [kv_cache_utils.py:715] GPU KV cache size: 81,920 tokens
INFO 06-21 04:52:40 [kv_cache_utils.py:719] Maximum concurrency for 1,024 tokens per request: 80.00x
INFO 06-21 04:52:40 [core.py:171] init engine (profile, create kv cache, warmup model) took 25.26 seconds
WARNING 06-21 04:52:42 [tokenizer.py:262] Using a slow tokenizer. This might cause a significant slowdown. Consider using a fast tokenizer instead.
WARNING 06-21 04:52:43 [core.py:101] Using configured V1 scheduler class vllm_ascend.core.scheduler.AscendScheduler. This scheduler interface is not public and compatibility may not be maintained.
Adding requests: 100%|██████| 2/2 [00:00<00:00, 907.66it/s]
Processed prompts: 0%| | 0/2 [00:00<?, ?it/s, est. speed input: 0.00 toks/s, output: 0.00 toks/s]
Processed prompts: 100%|██████| 2/2 [00:44<00:00, 22.10s/it, est. speed input: 0.27 toks/s, output: 0.36 toks/s]
Prompt: 'Hello, my name is', Generated text: ' Bob.Nice to meet you too,'
Prompt: 'The future of AI is', Generated text: ' not just about creating smarter machines but about'

</details>

[Yikun]

Accurarcy test passed: https://github.com/vllm-project/vllm-ascend/actions/runs/15795103338

lm-eval 0.4.9 was wrong results: - Qwen3 8B - V0: gsm8k 0.8324 ---> 0.6937 ceval 0.8187-->0.2816 - V1: gsm8k 0.8294 -->0.6929 ceval 0.2303-->0.2303

[1] https://github.com/vllm-project/vllm-ascend/actions/runs/15792250955
[2] (1 week ago) https://github.com/vllm-project/vllm-ascend/actions/runs/15647617544
[3] (revert 310p f7d6458) https://github.com/vllm-project/vllm-ascend/actions/runs/15793389183
[4] revert a558881 : https://github.com/vllm-project/vllm-ascend/actions/runs/15793693619
[5] revert pta ugprade 96fa7ff https://github.com/vllm-project/vllm-ascend/actions/runs/15794097953
[6] reset to 69b817e https://github.com/vllm-project/vllm-ascend/actions/runs/15794383868
[7] ab5d110 https://github.com/vllm-project/vllm-ascend/actions/runs/15794536328
[8] 94a52cf https://github.com/vllm-project/vllm-ascend/actions/runs/15794571309?pr=1344

[2] [8] same version!!!! ---> lm-evel version 0.4.8---->0.4.9

Let's see: https://github.com/vllm-project/vllm-ascend/actions/runs/15794812740

[Yikun]

# Start vLLM V0
export MODEL=Qwen/Qwen3-8B
export VLLM_USE_MODELSCOPE=true
VLLM_USE_MODELSCOPE=true python3 -m vllm.entrypoints.openai.api_server --model $MODEL \
         --tensor-parallel-size 1 --swap-space 16 --disable-log-stats \
         --disable-log-requests  --load-format dummy
# Start vLLM V1
export MODEL=Qwen/Qwen3-8B
export VLLM_USE_MODELSCOPE=true
VLLM_USE_V1=1 VLLM_USE_MODELSCOPE=true python3 -m vllm.entrypoints.openai.api_server --model $MODEL \
         --tensor-parallel-size 1 --swap-space 16 --disable-log-stats \
         --disable-log-requests  --load-format dummy
# Benchmark
export MODEL=Qwen/Qwen3-8B
export VLLM_USE_MODELSCOPE=true
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install -r /vllm-workspace/vllm-ascend/benchmarks/requirements-bench.txt
python3 /vllm-workspace/vllm/benchmarks/benchmark_serving.py --model $MODEL --dataset-name random \
         --random-input-len 200 --num-prompts 200 --request-rate 1 \
         --save-result --result-dir ./

Performance test:

https://docs.google.com/spreadsheets/d/1Z6KIp54n2NUhubMPImQrVtKnV5ZaUyu0FwP9NYtXdrA

[Yikun]

20250622 v0.9.1rc1

Run with A2:

A2 Download image

# First pull mirror to speedup
docker pull quay.nju.edu.cn/ascend/vllm-ascend:v0.9.1rc1
docker pull quay.io/ascend/vllm-ascend:v0.9.1rc1

export IMAGE=quay.io/ascend/vllm-ascend:v0.9.1rc1
docker run --rm \
        --name vllm-ascend \
        --device /dev/davinci4 \
        --device /dev/davinci5 \
        --device /dev/davinci6 \
        --device /dev/davinci7 \
        --device /dev/davinci_manager \
        --device /dev/devmm_svm \
        --device /dev/hisi_hdc \
        -v /usr/local/dcmi:/usr/local/dcmi \
        -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
        -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
        -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
        -v /etc/ascend_install.info:/etc/ascend_install.info \
        -v /root/.cache:/root/.cache \
        -p 8000:8000 \
        -it $IMAGE bash

# Make sure the commits are included:
cd /vllm-workspace/vllm-ascend/;git log -1;cd -

Run with 300I DUO:

300I DUO Download image

# First pull mirror to speedup
docker pull quay.nju.edu.cn/ascend/vllm-ascend:v0.9.1rc1-310p
docker pull quay.io/ascend/vllm-ascend:v0.9.1rc1-310p

export IMAGE=quay.io/ascend/vllm-ascend:v0.9.1rc1-310p
docker run --rm \
        --name vllm-ascend \
        --device /dev/davinci0 \
        --device /dev/davinci1 \
        --device /dev/davinci2 \
        --device /dev/davinci3 \
        --device /dev/davinci4 \
        --device /dev/davinci5 \
        --device /dev/davinci6 \
        --device /dev/davinci7 \
        --device /dev/davinci_manager \
        --device /dev/devmm_svm \
        --device /dev/hisi_hdc \
        -v /usr/local/dcmi:/usr/local/dcmi \
        -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
        -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
        -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
        -v /etc/ascend_install.info:/etc/ascend_install.info \
        -v /root/.cache:/root/.cache \
        -p 8000:8000 \
        -it $IMAGE bash

# Make sure the commits are included:
cd /vllm-workspace/vllm-ascend/;git log -1;cd -

API Test

export TP=4
VLLM_USE_V1=1 vllm serve /root/.cache/pangu-pro-moe-model \
    --tensor-parallel-size $TP \
    --swap-space 16 \
    --disable-log-stats  --disable-log-requests  \
    --trust-remote-code  --enforce-eager

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "/root/.cache/pangu-pro-moe-model",
        "prompt": "The future of AI is",
        "max_tokens": 128,
        "temperature": 0
    }'

Accuracy test

export TP=4
VLLM_USE_V1=1 vllm serve /root/.cache/pangu-pro-moe-model \
    --tensor-parallel-size $TP \
    --swap-space 16 \
    --disable-log-stats  --disable-log-requests  \
    --trust-remote-code  --enforce-eager

# Remove limit when real test
lm_eval \
  --model local-completions \
  --model_args model=/root/.cache/pangu-pro-moe-model,base_url=http://127.0.0.1:8000/v1/completions,tokenized_requests=False,trust_remote_code=True \
  --tasks gsm8k --batch_size auto --limit 10 \
  --output_path ./

Benchmark test

export TP=4
VLLM_USE_V1=1 vllm serve /root/.cache/pangu-pro-moe-model \
    --tensor-parallel-size $TP \
    --swap-space 16 \
    --disable-log-stats  --disable-log-requests  \
    --load-format dummy --trust-remote-code  --enforce-eager

# Benchmark
export MODEL=/root/.cache/pangu-pro-moe-model
export VLLM_USE_MODELSCOPE=true
pip config set global.index-url https://mirrors.tuna.tsinghua.edu.cn/pypi/web/simple
pip install -r /vllm-workspace/vllm-ascend/benchmarks/requirements-bench.txt
python3 /vllm-workspace/vllm/benchmarks/benchmark_serving.py --model $MODEL --dataset-name random \
         --random-input-len 200 --num-prompts 200 --request-rate 1 --trust_remote_code \
         --save-result --result-dir ./

A2 results:

============ Serving Benchmark Result ============
Successful requests:                     200
Benchmark duration (s):                  218.56
Total input tokens:                      39800
Total generated tokens:                  25600
Request throughput (req/s):              0.92
Output token throughput (tok/s):         117.13
Total Token throughput (tok/s):          299.24
---------------Time to First Token----------------
Mean TTFT (ms):                          283.86
Median TTFT (ms):                        266.48
P99 TTFT (ms):                           597.74
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          124.84
Median TPOT (ms):                        123.84
P99 TPOT (ms):                           134.04
---------------Inter-token Latency----------------
Mean ITL (ms):                           125.18
Median ITL (ms):                         123.62
P99 ITL (ms):                            255.52
==================================================

[Yikun]

export VLLM_USE_V1=1

vllm serve /home/pangu-pro-moe-mode/ \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--dtype "float16" \
--trust-remote-code \
--enforce-eager

import gc
from transformers import AutoTokenizer
import torch

from vllm import LLM, SamplingParams
from vllm.distributed.parallel_state import (destroy_distributed_environment,
                                             destroy_model_parallel)

def clean_up():
    destroy_model_parallel()
    destroy_distributed_environment()
    gc.collect()
    torch.npu.empty_cache()


if __name__ == "__main__":

    tokenizer = AutoTokenizer.from_pretrained("/home/pangu-pro-moe-mode/", trust_remote_code=True)
    tests = [
        "Hello, my name is",
        "The future of AI is",
    ]
    prompts = []
    for text in tests:
        messages = [
        {"role": "system", "content": ""},    # Optionally customize system content
        {"role": "user", "content": text}
    ]
        prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)        # 推荐使用官方的template
        prompts.append(prompt)
    sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=40)

    llm = LLM(model="/home/pangu-pro-moe-mode/",
            tensor_parallel_size=8,
            distributed_executor_backend="mp",
            enable_expert_parallel=True,
            dtype="float16",
            max_model_len=1024,
            trust_remote_code=True,
            enforce_eager=True)

    outputs = llm.generate(prompts, sampling_params)
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")

    del llm
    clean_up()