Feature Request: Add LLM Concurrency Limit with Customizable Model Grouping

[39499740]

What problem does this solve?

Many LLM providers enforce concurrency (RPM / concurrent request) limits on their APIs. Currently, Clawith has no built-in mechanism to control the number of simultaneous outgoing LLM requests. This leads to several problems:

1. No concurrency control – When multiple Agents are active simultaneously (e.g. via the Aware/Pulse Engine triggers, A2A conversations, or multi-channel IM messages), the system can fire an unbounded number of parallel LLM calls, easily exceeding a provider's rate limit and resulting in HTTP 429 errors or dropped connections.

2. Shared API key, shared limit – The current LLM configuration (PROVIDER_REGISTRY in llm_client.py) is keyed per-provider, not per-API-key. Multiple model entries (e.g. gpt-4o, gpt-4o-mini, o1-preview) may all use the same OpenAI API key and therefore share a single concurrency quota from the provider, but the system treats them independently with no awareness of this shared ceiling.

3. No way to group models – There is no facility to declare "these 5 models share a pool of N concurrent requests" or "these two providers should be merged into one limit group because they route through the same gateway."

4. Silent failures – When the provider rejects a request due to concurrency overflow, the user sees a generic LLMError rather than a queued/retried response.

Proposed solution

Introduce a Concurrency Limit Management layer with the following capabilities:

1. Per-group concurrency semaphores

Allow administrators to define named concurrency groups. Each group holds:

• A list of LLM config IDs (or provider + model patterns) that belong to the group
• A maximum concurrent request count (max_concurrency)

Internally this could be backed by asyncio.Semaphore instances keyed by group name.

2. Default auto-grouping by provider

Out of the box, group all models under the same provider (e.g. all openai models → one semaphore). Admins can override this by creating explicit custom groups.

3. Custom cross-provider grouping

Support merging models from different providers into a single group. This covers scenarios where:

• A reverse proxy / API gateway fronts multiple providers under one rate limit
• An organization uses a shared quota across multiple API keys
• Specific high-traffic models need a dedicated pool separate from the default

Example configuration concept:

concurrency_groups:
  - name: openai-shared
    max_concurrency: 5
    models: ["gpt-4o", "gpt-4o-mini", "o1-preview"]
  - name: deepseek-group
    max_concurrency: 3
    providers: ["deepseek"]
  - name: custom-gateway
    max_concurrency: 10
    models: ["claude-3-opus", "gpt-4o"]   # behind same proxy

4. Graceful queuing & visibility

• When a group is at capacity, new LLM requests wait in a queue (with configurable timeout) rather than failing immediately
• Expose current concurrency utilization via an admin API endpoint and optionally in the frontend dashboard
• Log warnings when approaching the limit (e.g. 80% utilization)

5. Admin UI

Add a concurrency management section in the enterprise settings page to configure groups without editing config files.

Why this is useful

• Reliability – Eliminates HTTP 429 errors from providers by respecting their limits
• Fairness – Ensures no single Agent or trigger monopolizes the LLM capacity
• Flexibility – Adapts to any deployment topology (single provider, multi-provider, gateway-proxied)
• Cost control – Prevents unexpected burst usage that could spike API costs
• Production readiness – Essential for multi-tenant SaaS deployments where many Agents run concurrently

Willing to contribute?

Yes – I would be interested in working on this.

[39499740]

@wisdomqin

Refined Design After Discussion

After further discussion, we have refined the design to be more user-friendly and avoid redundancy. Here is the updated proposal:

---

Core Principle

Concurrency limit always has exactly one source — either from the group the model belongs to, or from the model itself. Never both visible at the same time.

Model is in a group?  → Use the group's concurrency limit (model's own field is hidden)
Model is NOT in a group AND has max_concurrency? → Use the model's own limit
Model is NOT in a group AND max_concurrency is null? → No limit (backward compatible)

This avoids the confusion of having two places to configure the same thing.

---

Data Model Changes

llm_models table — add two fields:

Field	Type	Description
max_concurrency	int | null	Per-model concurrency limit. Only effective when model is NOT in a group. null = no limit.
concurrency_group_id	FK | null	References a concurrency group. A model can belong to at most one group.

New llm_concurrency_groups table:

Field	Type	Description
id	UUID	Primary key
tenant_id	UUID	Tenant isolation
name	str	User-defined group name (e.g. "OpenAI Production Pool")
max_concurrency	int	Shared concurrency limit for all models in this group
queue_timeout	int	Max seconds a request waits in queue before giving up (default: 30)
created_at	datetime	Timestamp

---

Frontend UX

Model Pool settings page — model list view:

Model State	UI Behavior
Not in a group	Model edit form shows max_concurrency field
In a group	Model edit form hides max_concurrency, shows read-only label: "Group: OpenAI Production Pool (limit: 5)"

New "Concurrency Groups" section in LLM tab:

1. "Create Group" button → opens a form with:
   • Group name (free text)
   • Max concurrency (number input)
   • Queue timeout in seconds (number input, default 30)
2. After creating a group, admin can add models via a popup dialog with a multi-select checklist
3. Each model appears in the checklist, already-grouped models are grayed out / disabled
4. Drag-and-drop or simple list to manage group membership
5. Delete group → all member models revert to using their own max_concurrency

---

Concurrency Control Logic

At the LLM call entry point (call_llm() in websocket.py, task_executor.py, trigger_daemon.py):

async def acquire_concurrency(model, agent):
    if model.concurrency_group_id:
        group = db.get(ConcurrencyGroup, model.concurrency_group_id)
        semaphore = get_or_create_group_semaphore(group.id, group.max_concurrency)
        timeout = group.queue_timeout
    elif model.max_concurrency:
        semaphore = get_or_create_model_semaphore(model.id, model.max_concurrency)
        timeout = 30  # default
    else:
        return  # no limit, proceed directly

    # Priority-aware acquire
    priority = agent_priority(agent)  # high=0, normal=1, low=2
    acquired = await wait_with_priority(semaphore, priority, timeout)
    if not acquired:
        raise ConcurrencyTimeoutError(model, timeout)

---

Agent Priority (3 tiers)

Add a priority field to Agent:

Value	Numeric	Behavior
high	0	First in queue — critical agents
normal	1	Default — most agents
low	2	Background/batch agents

When multiple requests are waiting for the same semaphore, higher-priority agents acquire first.

Frontend: simple dropdown in Agent settings, default normal.

---

Queue Status Feedback (WebSocket)

When a request is queued, immediately push status to the client:

{ "type": "queue_status", "position": 3, "estimated_wait_seconds": 12 }

When the request starts processing:

{ "type": "queue_status", "status": "processing" }

If queue timeout is reached:

{ "type": "error", "message": "Model is currently busy, please try again later" }

---

Smart Fallback

When an Agent has both primary_model_id and fallback_model_id:

1. Check primary model's concurrency availability
2. If primary is at capacity AND fallback exists AND fallback has capacity → use fallback immediately (no queue)
3. If both are at capacity → queue on primary model's group with agent priority

This prevents unnecessary waiting when a perfectly good fallback model is available.

---

Real-time Monitoring (Admin)

Add a concurrency dashboard in enterprise settings:

• Per-group current usage / limit (e.g. "3 / 5")
• Number of queued requests
• Historical peak usage (helps admins tune the limits)

---

Examples

Scenario	Configuration	Effect
Single GPT-4o model	Model max_concurrency=5, no group	Independent semaphore, max 5 concurrent
GPT-4o + GPT-4o-mini, same API key	Group "OpenAI" with max_concurrency=5, both models added	Shared semaphore, combined max 5
Local Ollama model	Model max_concurrency=null, no group	No concurrency limit
Claude + GPT-4o behind same proxy gateway	Group "Unified Gateway" with max_concurrency=10, both models added	Cross-provider shared limit

[lijiajun1997]

无脑加retry次数也行 发现有的code plan 压根不是按并发限制的 就是随机限制

[39499740]

无脑加retry次数也行 发现有的code plan 压根不是按并发限制的 就是随机限制

今天我用GLM的时候，发现GLM这边好像是并发限制的，我重复了很多次也没用

[wisdomqin]

Thanks for the detailed report and the well-thought-out design proposal — this is exactly the kind of contribution that helps us prioritize roadmap work. The two-phase approach (per-model semaphore first, then grouping) aligns well with our architecture.

Before we scope the implementation, we want to make sure we understand the full impact: besides GLM, are there other providers you're hitting rate limits on? (e.g. OpenAI, DeepSeek, Claude, Qwen, etc.)

Knowing which providers are most affected will help us prioritize the rollout order and default configurations.

[39499740]

Thanks for the prompt response!

To answer your question: at the moment I only have subscription plans for GLM and Minimax, and both of them enforce concurrency limits. I am not currently hitting rate limits on OpenAI, DeepSeek, Claude, Qwen, or other providers — but I suspect they would benefit from this feature as well if usage scales up.

So the immediate priority for me is GLM + Minimax concurrency control.

[lijiajun1997]

我用glm+jd+baidu+火山+minimax ，个人感知限流是百度>minimax＞JD＞火山≈GLM，不同家的限流策略也不一样 比如百度比较奇葩 ，1分钟1个请求和1分钟100个请求失败率是一样的，无脑叠加重试就行。目前测试按照claude code的逻辑重试10次，glm和火山基本可以正常持续运行，JD偶尔会超过10次报错断开。提交了一个PR把重试次数改成10次。 但是没有考虑多用户的情况的并发，建议考虑先用api聚合站做分流。