[Feature] [PD Disaggregation] simplify configuration for pd-disaggregated deployment, and refactor post-init and usage for all ports by liyonghua0910 · Pull Request #5415 · PaddlePaddle/FastDeploy

liyonghua0910 · 2025-12-06T05:17:21Z

Motivation

该 PR 旨在实现两个目标：

简化 PD 分离的部署流程和参数，包括端口号配置、RDMA 网卡检测、相关环境变量设置等等工序，实现【启动 Router】→【启动 P&D 实例】→【部署完成】的简易部署流程。其中启动参数的简化也期望适用于集中式部署和多 TP/DP 部署，并兼容通过 APIServer 和 MultiAPIServer 多 DP 服务的启动方式。
重构当前代码中与端口号相关的配置处理和使用逻辑。在参数初始化时，若用户未指定端口号，自动寻找可用端口，需要支持在线服务和离线接口；在多 DP 部署场景，在配置初始化时切分好各 DP 所需的端口号，而不是在使用时才临时切分。尽量实现配置的静态化、只读化，减少运行时的配置更改。

Modifications

ArgsUtils
- 新增 post_init_all_ports 参数后处理和检查流程，在 EngineArgs 初始化时，会检查用户传入的各类端口号数量是否正确。如果用户未传入端口号，会自动为用户分配所需数量的端口号。
FDConfig
- 去除旧的不规范的类型转换逻辑，在 config 初始化时会用 parse_ports() 方法统一将 *_port 类变量转成 list[int] 类型
- 🌟 新增 local_* 类端口号变量，包括 local_engine_worker_queue_port (int), local_rdma_comm_ports (list[int]), local_pd_comm_port (int), local_cache_queue_port (int)，在 DP/EP 场景用来指代当前 DP 使用的端口号，非 DP/EP 场景也统一使用 local_* 类的端口号变量
- 🌟 新增 postprocess_devices_and_ports 的 config 后处理流程，在 FDConfig.postprocess 中，会统一为 local_* 类端口号变量赋值，切分出当前 DP 所需的端口号，不建议在 config.py 以外的模块修改 FDConfig 对象
MultiAPIServer
- 新增参数检查流程，如果用户未传入端口号，会自动为用户分配所需数量的端口号；
- 如果用户传入的端口号数量不正确，会重新为用户分配所需数量的端口号
- 默认设置 FD_ENABLE_MULTI_API_SERVER 环境变量
Cache
- CacheTransferManager & CacheMessager 接收的参数名 engine_pid 修改为 ipc_suffix ，更贴合语义
- 🌟 RDMACacheTransfer 新增初始化代码，自动设置 KVCACHE_RDMA_NICS, KVCACHE_GDRCOPY_FLUSH_ENABLE 环境变量
CommonEngine
- 去除部分端口号列表切分、类型转换的逻辑（已经移到 config 层处理）
- 修改使用的 llm_logger 对象，如果是 DP 场景应该将日志写入 _dprank*.log 文件
Utils
- 新增端口号检测、解析和自动寻找可用端口号的工具函数
Examples
- 简化 start_v1_dp2.sh 和 start_v1_tp1.sh 的启动命令，新增用例 start_v1_tp2.sh，并优化 utils.sh 中的工具函数
Others
- 🌟 将所有端口号变量都配套修改为使用 local_* 类端口号变量
- 兼容使用 api server 启动多 DP 的方式
  - 去除部分 DP 逻辑的 EP 限制
  - 在 DP0 创建 DP1-N 时，深度拷贝当前 DP0 的 cfg 给各个 DP，避免 DP1-N 内部有修改 config 的操作互相干扰
  - 在 ExpertService 初始化时，根据 local_data_parallel_id 重写当前 DP 的部分配置
  - 🌟 让每个 DP 都创建一个 EngineCacheQueue 服务，而不是所有 DP 共享一个，与 EngineWorkerQueue 的架构对齐

Usage or Command

bash examples/splitwise/start_v1_dp2.sh

Accuracy Tests

$ bash examples/splitwise/start_v1_dp2.sh
ROUTER_PORT: 8274
nohup: redirecting stderr to stdout
P_SERVER_PORTS: 8629,8631
nohup: redirecting stderr to stdout
-------- WAIT FOR HEALTH --------
Port 8629: [OK]   200
Port 8631: [OK]   200
All services are ready!    [38s]
---------------------------------
D_SERVER_PORTS: 8534,8535
nohup: redirecting stderr to stdout
-------- WAIT FOR HEALTH --------
Port 8534: [OK]   200
Port 8535: [OK]   200
All services are ready!    [37s]
---------------------------------
{"id":"chatcmpl-3dcecc9a-cf59-45a1-876a-e66535c6ef23","object":"chat.completion","created":1765535525,"model":"/root/paddlejob/workspace/env_run/output/models/ERNIE-4.5-0.3B-Paddle/","choices":[{"index":0,"message":{"role":"assistant","content":"Hello! How can I assist you today?","multimodal_content":null,"reasoning_content":null,"audio_content":null,"tool_calls":null,"prompt_token_ids":null,"completion_token_ids":null,"prompt_tokens":null,"completion_tokens":null},"logprobs":null,"draft_logprobs":null,"prompt_logprobs":null,"finish_reason":"stop"}],"usage":{"prompt_tokens":8,"total_tokens":18,"completion_tokens":10,"prompt_tokens_details":{"cached_tokens":0,"image_tokens":0,"video_tokens":0},"completion_tokens_details":{"reasoning_tokens":0,"image_tokens":0}}}

Checklist

Add at least a tag in the PR title.
- Tag list: [[FDConfig],[APIServer],[Engine], [Scheduler], [PD Disaggregation], [Executor], [Graph Optimization], [Speculative Decoding], [RL], [Models], [Quantization], [Loader], [OP], [KVCache], [DataProcessor], [BugFix], [Docs], [CI], [Optimization], [Feature], [Benchmark], [Others], [XPU], [HPU], [GCU], [DCU], [Iluvatar], [Metax]]
- You can add new tags based on the PR content, but the semantics must be clear.
Format your code, run pre-commit before commit.
Add unit tests. Please write the reason in this PR if no unit tests.
Provide accuracy results.
If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

…factor post-init and usage for all ports

paddle-bot · 2025-12-06T05:17:27Z

Thanks for your contribution!

codecov-commenter · 2025-12-06T06:24:28Z

Codecov Report

❌ Patch coverage is 68.51852% with 85 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (develop@c9b47f9). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
fastdeploy/entrypoints/openai/multi_api_server.py	53.22%	14 Missing and 15 partials ⚠️
fastdeploy/engine/common_engine.py	58.33%	18 Missing and 2 partials ⚠️
fastdeploy/utils.py	64.51%	5 Missing and 6 partials ⚠️
...he_manager/transfer_factory/rdma_cache_transfer.py	77.77%	5 Missing and 3 partials ⚠️
fastdeploy/engine/args_utils.py	81.08%	4 Missing and 3 partials ⚠️
fastdeploy/engine/engine.py	42.85%	2 Missing and 2 partials ⚠️
fastdeploy/engine/expert_service.py	83.33%	0 Missing and 2 partials ⚠️
fastdeploy/entrypoints/openai/api_server.py	0.00%	2 Missing ⚠️
fastdeploy/worker/worker_process.py	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             develop    #5415   +/-   ##
==========================================
  Coverage           ?   62.08%           
==========================================
  Files              ?      329           
  Lines              ?    41287           
  Branches           ?     6295           
==========================================
  Hits               ?    25633           
  Misses             ?    13701           
  Partials           ?     1953

Flag	Coverage Δ
GPU	`62.08% <68.51%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot

Pull request overview

This PR refactors port configuration for PD-disaggregated deployments by introducing automatic port allocation and consolidating port management logic. The changes aim to simplify configuration by removing the need for users to manually specify multiple ports per service.

Key Changes

Introduced automatic port discovery via find_free_ports() function with validation in post_init_all_ports()
Centralized port extraction logic in FDConfig.postprocess() to set per-DP ports (local_engine_worker_queue_port, etc.)
Added RDMA environment auto-detection via new get_rdma_nics.sh script

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 15 comments.

Show a summary per file

File	Description
fastdeploy/utils.py	Added `parse_ports()`, `find_free_ports()`, and modified `is_port_available()` for port management
fastdeploy/engine/args_utils.py	Added `post_init_all_ports()` to automatically allocate ports; changed default `cache_transfer_protocol` to "ipc,rdma"
fastdeploy/config.py	Moved port extraction logic to `postprocess()` method; added `local_engine_worker_queue_port` attribute
fastdeploy/worker/worker_process.py	Moved port assignment from pre-config to post-config initialization
fastdeploy/engine/engine.py	Updated to use `local_engine_worker_queue_port` for DP queue services
fastdeploy/engine/common_engine.py	Simplified port handling by using `local_engine_worker_queue_port` directly
fastdeploy/engine/expert_service.py	Removed local port/device ID extraction logic (now handled in config)
fastdeploy/splitwise/splitwise_connector.py	Changed to use scalar `pd_comm_port` and `local_device_ids`
fastdeploy/cache_manager/prefix_cache_manager.py	Renamed `pid_suffix` to `ipc_suffix`; updated RDMA port usage
fastdeploy/cache_manager/cache_transfer_manager.py	Renamed `engine_pid` to `ipc_suffix` for clarity
fastdeploy/cache_manager/cache_messager.py	Renamed `engine_pid` to `ipc_suffix` for consistency
fastdeploy/cache_manager/transfer_factory/rdma_cache_transfer.py	Added automatic RDMA environment setup and NIC detection
fastdeploy/cache_manager/transfer_factory/get_rdma_nics.sh	New script for detecting RDMA-capable network interfaces
examples/splitwise/utils.sh	Improved health check display with UDP port checking
examples/splitwise/start_v1_tp1.sh	Simplified example by removing manual port specifications

…_pd_config_1206

…s ci test

Copilot

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 16 comments.

Comments suppressed due to low confidence (1)

fastdeploy/engine/args_utils.py:557

This comment appears to contain commented-out code.

            # for port in ports:
            #     assert is_port_available("0.0.0.0", port), f"Parameter `{name}`:{port} is already in use."

…_pd_config_1206

juncaipeng

LGTM

…_pd_config_1206

…ated deployment, and refactor post-init and usage for all ports (PaddlePaddle#5415) * [feat] simplify configuration for pd-disaggregated deployment, and refactor post-init and usage for all ports * [fix] fix some bugs * [fix] fix rdma port for cache manager/messager * [fix] temporarily cancel port availability check to see if it can pass ci test * [feat] simplify args for multi api server * [fix] fix dp * [fix] fix port for xpu * [fix] add tests for ports post processing & fix ci * [test] fix test_multi_api_server * [fix] fix rdma_comm_ports args for multi_api_server * [fix] fix test_common_engine * [fix] fix test_cache_transfer_manager * [chore] automatically setting FD_ENABLE_MULTI_API_SERVER * [fix] avoid api server from creating engine_args twice * [fix] fix test_run_batch * [fix] fix test_metrics * [fix] fix splitwise connector init * [test] add test_rdma_transfer and test_expert_service * [fix] fix code syntax * [fix] fix test_rdma_transfer and build wheel with rdma script

[feat] simplify configuration for pd-disaggregated deployment, and re…

209bce3

…factor post-init and usage for all ports

Jiang-Jia-Jun requested a review from Copilot December 8, 2025 03:40

Copilot started reviewing on behalf of Jiang-Jia-Jun December 8, 2025 03:40 View session

Copilot AI reviewed Dec 8, 2025

View reviewed changes

liyonghua0910 added 8 commits December 9, 2025 11:34

[fix] fix some bugs

29a0ee5

Merge remote-tracking branch 'upstream/develop' into develop+simplify…

9d216c0

…_pd_config_1206

[fix] fix rdma port for cache manager/messager

47a86a8

[fix] temporarily cancel port availability check to see if it can pas…

f07c2e0

…s ci test

[feat] simplify args for multi api server

cb2e618

[fix] fix dp

9c9a660

[fix] fix port for xpu

b3fec44

[fix] add tests for ports post processing & fix ci

8ceb83c

juncaipeng requested a review from Copilot December 10, 2025 13:55

Copilot started reviewing on behalf of juncaipeng December 10, 2025 13:55 View session

Copilot AI reviewed Dec 10, 2025

View reviewed changes

liyonghua0910 added 10 commits December 11, 2025 12:29

[test] fix test_multi_api_server

c2bc5d6

[fix] fix rdma_comm_ports args for multi_api_server

32fa9e8

[fix] fix test_common_engine

d1ab65c

[fix] fix test_cache_transfer_manager

1832afb

[chore] automatically setting FD_ENABLE_MULTI_API_SERVER

d520838

Merge remote-tracking branch 'upstream/develop' into develop+simplify…

555044f

…_pd_config_1206

[fix] avoid api server from creating engine_args twice

624d02f

[fix] fix test_run_batch

d164731

[fix] fix test_metrics

1fb86eb

[fix] fix splitwise connector init

7d4ff93

juncaipeng previously approved these changes Dec 15, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/develop' into develop+simplify…

46ad556

…_pd_config_1206

liyonghua0910 dismissed juncaipeng’s stale review via 46ad556 December 15, 2025 06:25

liyonghua0910 requested a review from jiangjiajun December 15, 2025 06:26

liyonghua0910 added 3 commits December 15, 2025 16:39

[test] add test_rdma_transfer and test_expert_service

556c7b3

[fix] fix code syntax

e83f806

[fix] fix test_rdma_transfer and build wheel with rdma script

3e6fe8b

Jiang-Jia-Jun previously approved these changes Dec 16, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/develop' into develop+simplify…

7857a75

…_pd_config_1206

liyonghua0910 dismissed Jiang-Jia-Jun’s stale review via 7857a75 December 16, 2025 06:42

Jiang-Jia-Jun approved these changes Dec 16, 2025

View reviewed changes

Jiang-Jia-Jun merged commit 0c8c636 into PaddlePaddle:develop Dec 17, 2025
16 of 18 checks passed

Conversation

liyonghua0910 commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Modifications

Usage or Command

Accuracy Tests

Checklist

Uh oh!

paddle-bot Bot commented Dec 6, 2025

Uh oh!

codecov-commenter commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Key Changes

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

juncaipeng left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

liyonghua0910 commented Dec 6, 2025 •

edited

Loading

codecov-commenter commented Dec 6, 2025 •

edited

Loading