xpay treats a local "Too many HTLCs" failure as a 1 msat capacity shortfall and floods the route with dust parts.

When our own outgoing channel has no free HTLC slots, channeld fails the HTLC with temporary_channel_failure ("Too many HTLCs"). xpay handles that like a liquidity failure and tells askrene the channel is constrained to the attempted amount. askrene stores that as "amount minus 1 msat". The same channel stays the cheapest route, so the next getroutes sends the big part over it again, 1 msat smaller, and routes the 1 msat leftover elsewhere. Each retry fails again on the full slots, so xpay produces one new 1–17 msat part per failure. A slot shortage doesn't depend on the amount, so this loop can never succeed.

Observed (mainnet, v26.06.8)

Payment of 298,495 sat, retry_for 60 s:

From the first "Too many HTLCs" on our own channel (erring_index 0), xpay ran this cycle 98 times in 14
s, about once every 60 ms.
The dust parts summed to 237 msat. They went over other channels, reached the destination, and were held
there for about 2 minutes until mpp_timeout.
The main part (~295.5k sat) never left our node, so the payment failed.

xpay log (ids shortened):

      260: routes for 295521663msat = [1msat via 4 hops, 295521662msat via 2 hops]
      260: ...: We got failed: WIRE_TEMPORARY_CHANNEL_FAILURE (WIRE_TEMPORARY_CHANNEL_FAILURE: Too many HTLCs)
  for <local scidd>, assuming it can't carry 295521662msat
      260: routes for 295521662msat = [1msat via 4 hops, 295521661msat via 2 hops]
      260: ...: We got failed: WIRE_TEMPORARY_CHANNEL_FAILURE (... Too many HTLCs) for <local scidd>, assuming
  it can't carry 295521661msat
      ... (98 iterations)
      260: getroutes_done: need more (was_routing 295521655msat, needs_routing 295521656msat)

According to my LLM:

Where it happens

channeld reports a full slot count as a generic temporary_channel_failure:
channeld/channeld.c#L6602-L6605. The count check, including our own limit applied to
locally added HTLCs: channeld/full_channel.c#L728-L740
For a local failure (index == 0), xpay has the precise reason ("Too many HTLCs") in the error message:
plugins/xpay/xpay.c#L1008-L1013
It still maps every temporary_channel_failure to "can't carry this amount":
plugins/xpay/xpay.c#L1125-L1131
and sends askrene-inform-channel with inform: constrained and the attempted amount:
plugins/xpay/xpay.c#L1181-L1193
askrene turns that into a maximum of amount − 1 msat ("we were one msat short"):
plugins/askrene/askrene.c#L1199-L1211

@Lagrang3

@Lagrang3

Production confirmation from an OCEAN payout node running CLN v26.06.9. This looks like the same failure mode.

Configuration:

xpay-handle-pay=true
xpay-slow-mode=false

The application caller is sequential: it loops over payout items and awaits each pay RPC before starting the next item. The relevant shape is:

for payout in payouts {
    dispatch_payout(..., &mut payout).await;
}

// dispatch_payout
let pay_result = node.pay(payout).await;

Our wrapper ultimately makes a synchronous pay RPC, which xpay intercepts. Thus the part concurrency below is inside xpay, not concurrent application calls.

A sanitized production excerpt from 2026-10-09 (SCID redacted) shows the same local channel failing repeatedly with widely varying main/dust-sized amounts in the same millisecond:

2026-10-09T14:37:11.290Z xpay: We got failed: WIRE_TEMPORARY_CHANNEL_FAILURE
  (WIRE_TEMPORARY_CHANNEL_FAILURE: Too many HTLCs) for <local-scid>/0,
  assuming it can't carry 2677145msat
2026-10-09T14:37:11.290Z xpay: ... can't carry 9754msat
2026-10-09T14:37:11.290Z xpay: ... can't carry 96290msat
2026-10-09T14:37:11.290Z xpay: ... can't carry 6838msat
2026-10-09T14:37:11.290Z xpay: ... can't carry 9758msat
2026-10-09T14:37:11.290Z xpay: ... can't carry 23355msat
2026-10-09T14:37:11.290Z xpay: ... can't carry 2918msat
2026-10-09T14:37:11.291Z xpay: ... can't carry 3933msat
2026-10-09T14:38:26.114Z xpay: Timed out after after 44 attempts. We got failed:
  WIRE_TEMPORARY_CHANNEL_FAILURE
  (WIRE_TEMPORARY_CHANNEL_FAILURE: Too many HTLCs) for <local-scid>/0 ...

listsendpays for that single payment hash (hash redacted):

created range:        2026-10-09T14:37:10Z .. 14:37:50Z
attempt rows:         44
unique partids:       44
max simultaneous:     33
final statuses:       44 failed
parts started/sec:    13 at :32, 16 at :33 (top seconds)

The affected channel advertises max_accepted_htlcs=30.

A payment immediately before it showed the same pattern:

attempt rows:         50
unique partids:       50
max simultaneous:     32
parts started at once: 30 in one second
final statuses:       50 failed

Across the bounded mature-payout window, the largest single payment reached 70 simultaneously live HTLC attempt rows for one payment hash. This is based on listsendpays created_at/completed_at intervals, not the number of sequential application RPC calls.

This also explains why adding sat liquidity would not solve the observed failure: the channels had ample spendable balance, but xpay exhausted HTLC slots and then treated the local slot failure as an amount constraint, producing additional parts.

No invoices, recipient identifiers, or real payment hashes are included here. I can provide a more focused debug trace privately if useful.

This is one issue this PR #9380 solves.

xpay: don't treat a full HTLC slot table as a capacity shortfall

@Lagrang3

Nice, @Lagrang3 do you have any eta on it?

Navigation Menu

Repository navigation

xpay treats a local "Too many HTLCs" failure as a 1 msat capacity shortfall and floods the route with dust parts. #9597

Observed (mainnet, v26.06.8)

Where it happens

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

xpay treats a local "Too many HTLCs" failure as a 1 msat capacity shortfall and floods the route with dust parts. #9597

Description

Observed (mainnet, v26.06.8)

Where it happens

Activity

daywalker90 commented on Oct 2, 2026

vincenzopalazzo commented on Oct 9, 2026

Lagrang3 commented on Oct 12, 2026

vincenzopalazzo commented on Oct 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions