Skip to content

Raise a clear error for QLoRA + vllm_mode="server" instead of a cryptic vLLM crash#6440

Open
DaoyuanLi2816 wants to merge 1 commit into
huggingface:mainfrom
DaoyuanLi2816:fix-qlora-vllm-server-mode-guard
Open

Raise a clear error for QLoRA + vllm_mode="server" instead of a cryptic vLLM crash#6440
DaoyuanLi2816 wants to merge 1 commit into
huggingface:mainfrom
DaoyuanLi2816:fix-qlora-vllm-server-mode-guard

Conversation

@DaoyuanLi2816

@DaoyuanLi2816 DaoyuanLi2816 commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

Rejects QLoRA with vllm_mode="server" before weight synchronization. Merging a 4-bit PEFT adapter dequantizes its weights, which cannot be loaded by a server expecting packed 4-bit tensors.

The same guard is applied to VLLMGeneration and the self-contained OnlineDPOTrainer. The error directs users to an unquantized server or colocate mode. This is a fail-fast fix for #4973, not full quantized-server support.

Tests

  • server-mode guard tests for both implementations (3 passed, 1 skipped without local 4-bit support)
  • Ruff check and format check on the changed files

AI writing disclosure

  • AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.

Comment thread trl/generation/vllm_generation.py
@DaoyuanLi2816
DaoyuanLi2816 force-pushed the fix-qlora-vllm-server-mode-guard branch from fc71998 to e597239 Compare July 24, 2026 06:53

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e597239. Configure here.

Comment thread tests/experimental/test_online_dpo_trainer.py
@DaoyuanLi2816
DaoyuanLi2816 force-pushed the fix-qlora-vllm-server-mode-guard branch from e597239 to ddf0907 Compare July 24, 2026 07:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant