[FEATURE] Migrate and Enhance Adaptive Service Throttling in dubbo-go

[nagisa-kunhah]

Description

Fixes #3336

Progress:

• Review the current adaptive throttling capability in dubbo-go
• Run pressure tests to understand its capability limits
• Move the adaptive service plugin to dubbo-go-extensions
• Add dubbo-go-samples for adaptive service

Pressure tests

Test scenario

• samples/adaptive_service/protect_provider/{server,client}: verifies provider protection under high client concurrency by tracking rejects and server-side max active requests.
• samples/adaptive_service/rtt_shrink/{server,client}: verifies limiter behavior across fast/medium/slow RTT stages and records limitation/remaining/inflight changes.
• samples/adaptive_service/p2c_healthy/{server,client}: verifies multi-provider adaptive P2C routing by comparing per-provider hit ratio and remaining capacity.

Test results

✅ protect_provider：

• Config: 200 client concurrency, 200ms provider handler delay, 30s duration.
• Result: client sent 243,940 requests in total; 222,720 were rejected by adaptive service, failed=0, reject rate was about 91%. The provider business handler only processed about 21,020 requests, and rejected requests did not enter the handler.
• Conclusion: provider-side adaptive throttling can shed overload traffic before the business handler and keeps unexpected RPC failures at zero under high concurrency.

⚠️ rtt_shrink：

• Config: 200 client concurrency, staged provider handler delay fast:20ms:30s, medium:100ms:20s, slow:500ms:40s, 90s duration.
• Result: the staged delay switch worked as expected. Client-side latency increased with each phase: the fast phase stayed around tens of milliseconds, the medium phase rose to about 100ms, and the slow phase rose to about 500ms. failed=0 throughout the run. The limiter was found and reported continuously by the provider stats endpoint. During the fast phase, limiter_limitation grew from about 55 to a peak of about 123. However, after RTT increased to the 500ms slow phase, limiter_limitation stayed around 122 and did not drop meaningfully below the fast-phase peak. Rejections increased continuously under the high offered load, reaching about 888,000 total rejects by the end of the run.

✅ p2c_healthy：

• Config: three providers with different handler delays: fast=20ms, medium=100ms, slow=300ms; 200 client concurrency; 90s duration; adaptive cluster + P2C load balancing enabled.
• Result: P2C quickly avoided the slow provider after warmup. The final cumulative traffic distribution was about fast=44%, medium=54%, and slow=1.8%, with failed=0. The slow provider's interval traffic dropped to 0 later in the test. The medium provider was repeatedly selected as the healthiest node because its reported remaining capacity was higher; it reached limiter_limitation=500, while the fast provider stayed around 200.
• Conclusion: P2C can use adaptive service metrics to bias traffic away from unhealthy providers and toward nodes with higher remaining capacity. One caveat is that the 100ms provider received more traffic than the 20ms provider, which points to a possible HillClimbing expansion/parameter issue rather than a P2C selection failure.

Checklist

• I confirm the target branch is develop
• Code has passed local testing
• I have added tests that prove my fix is effective or that my feature works

[codecov-commenter]

Codecov Report

❌ Patch coverage is 1.74772% with 1293 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.11%. Comparing base (60d1c2a) to head (1982c6b).
⚠️ Report is 973 commits behind head on develop.

Files with missing lines	Patch %	Lines
...e_test/adaptive_service/p2c_healthy/client/main.go	0.00%	370 Missing ⚠️
...ee_test/adaptive_service/rtt_shrink/client/main.go	0.00%	279 Missing ⚠️
...t/adaptive_service/protect_provider/client/main.go	0.00%	264 Missing ⚠️
...ee_test/adaptive_service/rtt_shrink/server/main.go	0.00%	127 Missing ⚠️
...ptive_service/protect_provider/proto/protect.pb.go	0.00%	78 Missing ⚠️
...e_test/adaptive_service/p2c_healthy/server/main.go	0.00%	75 Missing ⚠️
...t/adaptive_service/protect_provider/server/main.go	0.00%	69 Missing ⚠️
...e_service/protect_provider/proto/protect.triple.go	0.00%	31 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3347      +/-   ##
===========================================
+ Coverage    46.76%   55.11%   +8.34%     
===========================================
  Files          295      486     +191     
  Lines        17172    39951   +22779     
===========================================
+ Hits          8031    22019   +13988     
- Misses        8287    16218    +7931     
- Partials       854     1714     +860     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Alanxtl]

可以参考uber的实现https://www.infoq.com/news/2024/02/uber-dynamic-load-shedding/?utm_source=email&utm_medium=editorial&utm_campaign=SpecialNL&utm_content=02292024&forceSponsorshipId=58a6b10a-7b64-4cfd-a08d-c065e2458967

[nagisa-kunhah]

可以参考uber的实现https://www.infoq.com/news/2024/02/uber-dynamic-load-shedding/?utm_source=email&utm_medium=editorial&utm_campaign=SpecialNL&utm_content=02292024&forceSponsorshipId=58a6b10a-7b64-4cfd-a08d-c065e2458967

@Alanxtl hello，也麻烦看下，pr description里有说到，我测rtt_shrink这个case的时候发现，当延迟升高的时候（20ms升高到100ms，再升高到500ms），limiter对inflight的限制似乎并没有明显的减少，而是维持在原来的水平，不清楚这个是否符合原来的预期？测试用的代码放在presee_test/adaptive_service/rtt_shrink下。

[Alanxtl]

可以参考uber的实现https://www.infoq.com/news/2024/02/uber-dynamic-load-shedding/?utm_source=email&utm_medium=editorial&utm_campaign=SpecialNL&utm_content=02292024&forceSponsorshipId=58a6b10a-7b64-4cfd-a08d-c065e2458967

@Alanxtl hello，也麻烦看下，pr description里有说到，我测rtt_shrink这个case的时候发现，当延迟升高的时候（20ms升高到100ms，再升高到500ms），limiter对inflight的限制似乎并没有明显的减少，而是维持在原来的水平，不清楚这个是否符合原来的预期？测试用的代码放在presee_test/adaptive_service/rtt_shrink下。

这更像是暴露了当前 HillClimbing 实现的“不敏感/参数问题”，不太应该当成完全符合预期。

代码原因大概在这里：

• hill_climbing.go：limiter 每个 update round 才基于 transactionNum/rttAvg 计算 maxCapacity、tps，不是 RTT 一升高就立即降并发。
• hill_climbing.go：收缩条件要求 bestMaxCapacity - maxCapacity 和 RTT 劣化同时满足硬编码阈值。
• hill_climbing.go：真正 shrink 时也不是按 RTT 比例降低，而是回到 bestLimitation - log(limitation) 附近，所以下降幅度可能很小。
• rtt_shrink/server/main.go：压测确实是通过服务端 Sleep(currentStage.delay) 人为拉高 handler RTT。
• rtt_shrink/server/main.go：观测的 limiter_limitation 是直接从 provider 侧 limiter snapshot 暴露出来的，不是客户端自己估出来的。

这个可能是已知限制。预期上 adaptive concurrency 应该在 RTT 明显恶化、吞吐不再提升时收缩；但当前算法受历史 best metrics、硬编码阈值、update interval 和 shrink 幅度影响，在 20ms -> 500ms 的阶梯压测下没有明显降下来。

另外uber的那个实现太过于复杂了，参考一下就行，不用实现

[sonarqubecloud]

Quality Gate passed

Issues
22 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
23.7% Duplication on New Code

See analysis details on SonarQube Cloud

[Alanxtl]

hi @nagisa-kunhah , still working on this?

[nagisa-kunhah]

hi @nagisa-kunhah , still working on this?

Hi @Alanxtl , thanks for your reminder. I will update it this week.

[nagisa-kunhah]

Hi @Alanxtl , 我简单整理了一下，对当前 dubbo-go 里 Adaptive Service Throttling 相关代码做了一轮梳理。

因为各个模块联系还比较大，我就以server和client两边为分界梳理了两个文档：

• server 侧文档：filter/adaptivesvc/server.md
• client 侧文档：filter/adaptivesvc/client.md

目前文档里主要覆盖了这些内容：

• provider 侧 adaptive service filter
• method-level limiter
• Hill Climbing limiter
• consumer 侧 adaptiveService cluster
• P2C 如何使用 provider 返回的 adaptive-service.remaining
• 当前配置入口和使用方式
• 主要调用链
• 涉及的 attachments / attributes
• server / client 两侧各自的初始化和运行时流程

关于迁移到 dubbo-go-extensions，我建议迁移实现代码，core 继续保留接口、配置入口和字符串 key。这样依赖方向会和现有 extensions 的模式一致。extension 依赖 dubbo-go core 的接口和注册表，dubbo-go core 不反向依赖 extension。

建议迁移到 dubbo-go-extensions 的模块：

• filter/adaptivesvc：provider 侧 adaptive service filter，实现 padasvc filter 的注册和请求处理。
• filter/adaptivesvc/limiter：provider 侧 method-level limiter，包括 Hill Climbing limiter。
• cluster/cluster/adaptivesvc：consumer 侧 adaptiveService cluster，负责选择 provider、写入 adaptive-service.enabled、读取 adaptive-service.remaining 并更新本地 metrics。
• cluster/loadbalance/p2c：P2C load balancer。当前 adaptiveService cluster 强制使用 p2c，并且 P2C 当前读取的是 metrics.HillClimbing 这个 adaptive service remaining 指标，所以和 adaptive service 一起迁移更完整。
• cluster/metrics：consumer 侧保存 provider remaining 的本地 metrics。adaptiveService cluster 写入这里，P2C 从这里读取。

这些模块迁到 extensions 后，extensions 可以提供一个聚合 imports 包，例如：

import (
	_ "github.com/apache/dubbo-go-extensions/imports/adaptivesvc"
)

这个 imports 包负责 blank import 上面的实现模块，把这些实现注册到 extension registry：

• extension.SetFilter("padasvc", ...)
• extension.SetCluster("adaptiveService", ...)
• extension.SetLoadbalance("p2c", ...)

建议继续留在 dubbo-go core 的部分：

• common/constant 里的字符串 key：例如 ClusterKeyAdaptiveService、LoadBalanceKeyP2C、AdaptiveServiceProviderFilterKey、AdaptiveServiceEnabledKey、AdaptiveServiceRemainingKey、AdaptiveServiceInflightKey。保留这些 key 可以减少兼容性影响。
• 配置字段：consumer.adaptive-service、provider.adaptive-service、provider.adaptive-service-verbose 可以继续保留在 core。
• 配置自动写入逻辑：consumer 侧继续把 adaptive-service 转成 cluster=adaptiveService / loadbalance=p2c，provider 侧继续把 adaptive-service 转成 service.filter=padasvc。
• 公开 options：例如 WithClusterAdaptiveService()、WithClientClusterAdaptiveService()、WithLoadBalanceP2C()、WithClientLoadBalanceP2C()、WithServerAdaptiveService()、WithServerAdaptiveServiceVerbose()。这些 option 只设置字符串 key，可以继续留在 core。

需要调整的是注册入口：imports/imports.go 不应该 import dubbo-go-extensions，否则会形成 dubbo-go -> dubbo-go-extensions -> dubbo-go 的 module 依赖环。迁移后用户需要额外引入 extensions 的 imports 包来注册实现，例如同时引入：

import (
	_ "dubbo.apache.org/dubbo-go/v3/imports"
	_ "github.com/apache/dubbo-go-extensions/imports/adaptivesvc"
)

这样 core 继续负责配置和 key，extensions 负责实现和注册，用户侧通过 imports 提供可被配置引用的实现。

另外，提到的优化，我们可以在迁移过程中进行优化和测试，下周可以开始迁移。

[AlexStocks]

@nagisa-kunhah hello, do u have time to complete this task?

[nagisa-kunhah]

@nagisa-kunhah hello, do u have time to complete this task?

@AlexStocks ok, this pr is just to upload the pressure tests. The rest of work will progress on dubbo-go-extensions repo.

[AlexStocks]

@nagisa-kunhah hello, do u have time to complete this task?

@AlexStocks ok, this pr is just to upload the pressure tests. The rest of work will progress on dubbo-go-extensions repo.

so many file conflictions.

[sonarqubecloud]

Quality Gate passed

Issues
13 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
28.7% Duplication on New Code

See analysis details on SonarQube Cloud