Skip to content

[Bug] 0.14.0版本能否支持对Qwen3.5系列模型的量化？ #4720

Open
Open
[Bug] 0.14.0版本能否支持对Qwen3.5系列模型的量化？#4720
Assignees
lvhan028

Description

@legends-7
legends-7
opened on Jun 30, 2026

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

非常感谢LMdepoly开源框架对LLM领域所作的贡献，最近关注到0.14.0版本新增对Qwen3.5系列模型的支持，于是迫不及待想体验其新特性，但是我在使用LMdeploy的量化命令对Qwen3.5-4B模型进行量化的时候出现了错误，请问是框架暂不支持对Qwen3.5系列模型的量化功能还是我这边存在操作错误？

Reproduction

lmdeploy lite auto_awq models/Qwen3.5-4B --work-dir models/Qwen3.5-4B-w4a16 --calib-dataset "openwebtext"

Environment

sys.platform: linux
Python: 3.12.13 (main, Mar  4 2026, 09:23:07) [GCC 11.4.0]
CUDA available: True
MUSA available: False
numpy_random_seed: 2147483648
GPU 0: NVIDIA RTX A4500
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.8, V12.8.93
GCC: x86_64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
PyTorch: 2.10.0+cu128
PyTorch compiling details: PyTorch built with:
  - GCC 13.3
  - C++ Version: 201703
  - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - LAPACK is enabled (usually provided by MKL)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 12.8
  - NVCC architecture flags: -gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90;-gencode;arch=compute_100,code=sm_100;-gencode;arch=compute_120,code=sm_120
  - CuDNN 91.0.2  (built against CUDA 12.9)
  - Magma 2.6.1
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=449b1768410104d3ed79d3bcfe4ba1d65c7f22c0, CUDA_VERSION=12.8, CUDNN_VERSION=9.10.2, CXX_COMPILER=/opt/rh/gcc-toolset-13/root/usr/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_FBGEMM_GENAI -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-dangling-reference -Wno-error=dangling-reference -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.10.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF,

TorchVision: 0.25.0+cu128
LMDeploy: 0.14.0+
transformers: 5.12.1
fastapi: 0.138.0
pydantic: 2.13.4
triton: 3.6.0
NVIDIA Topology:
        GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-15    0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Error traceback

2026-06-29 10:14:46,683 - lmdeploy - INFO - builder.py:75 - matching vision model: Qwen3_5Model
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 723/723 [00:00<00:00, 15195.46it/s]
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
Move model.visual to GPU.
Move model.language_model.embed_tokens to GPU.
Move model.language_model.layers.0 to CPU.
Move model.language_model.layers.1 to CPU.
Move model.language_model.layers.2 to CPU.
Move model.language_model.layers.3 to CPU.
Move model.language_model.layers.4 to CPU.
Move model.language_model.layers.5 to CPU.
Move model.language_model.layers.6 to CPU.
Move model.language_model.layers.7 to CPU.
Move model.language_model.layers.8 to CPU.
Move model.language_model.layers.9 to CPU.
Move model.language_model.layers.10 to CPU.
Move model.language_model.layers.11 to CPU.
Move model.language_model.layers.12 to CPU.
Move model.language_model.layers.13 to CPU.
Move model.language_model.layers.14 to CPU.
Move model.language_model.layers.15 to CPU.
Move model.language_model.layers.16 to CPU.
Move model.language_model.layers.17 to CPU.
Move model.language_model.layers.18 to CPU.
Move model.language_model.layers.19 to CPU.
Move model.language_model.layers.20 to CPU.
Move model.language_model.layers.21 to CPU.
Move model.language_model.layers.22 to CPU.
Move model.language_model.layers.23 to CPU.
Move model.language_model.layers.24 to CPU.
Move model.language_model.layers.25 to CPU.
Move model.language_model.layers.26 to CPU.
Move model.language_model.layers.27 to CPU.
Move model.language_model.layers.28 to CPU.
Move model.language_model.layers.29 to CPU.
Move model.language_model.layers.30 to CPU.
Move model.language_model.layers.31 to CPU.
Move model.language_model.norm to GPU.
Move model.language_model.rotary_emb to GPU.
Move lm_head to GPU.
Loading calibrate dataset ...
README.md: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7.46k/7.46k [00:00<00:00, 11.7MB/s]
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
train-00000-of-00080.parquet: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 303M/303M [01:19<00:00, 3.79MB/s]
Generating train split:   1%|██▎                                                                                                                                                                                    | 100173/8013769 [00:00<01:11, 111280.21 examples/s]
Map: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30000/30000 [00:51<00:00, 582.67 examples/s]
model.language_model.layers.0, samples: 128, max gpu memory: 5.84 GB
model.language_model.layers.1, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.2, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.3, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.4, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.5, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.6, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.7, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.8, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.9, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.10, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.11, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.12, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.13, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.14, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.15, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.16, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.17, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.18, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.19, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.20, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.21, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.22, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.23, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.24, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.25, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.26, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.27, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.28, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.29, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.30, samples: 128, max gpu memory: 7.09 GB
model.language_model.layers.31, samples: 128, max gpu memory: 7.09 GB
Traceback (most recent call last):
  File "/opt/py3/bin/lmdeploy", line 6, in <module>
    sys.exit(run())
             ^^^^^
  File "/opt/py3/lib/python3.12/site-packages/lmdeploy/cli/entrypoint.py", line 39, in run
    args.run(args)
  File "/opt/py3/lib/python3.12/site-packages/lmdeploy/cli/lite.py", line 115, in auto_awq
    auto_awq(**kwargs)
  File "/opt/py3/lib/python3.12/site-packages/lmdeploy/lite/apis/auto_awq.py", line 119, in auto_awq
    fc2fcs = FC_FCS_MAP[layer_type]
             ~~~~~~~~~~^^^^^^^^^^^^
KeyError: 'Qwen3_5DecoderLayer'

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions