[bug]: Consider failing HTLC backwards before upstream claims on-chain

[BhaagBoseDK]

Background

Consider an HTLC chain

Peer A -> Peer B -> Offline Peer

And assume Peer B Force Closes on Offline Peer due to HTLC missing in remote commitment upon expiry of HTLC.

The Force Close transaction is stuck in mempool for 144 blocks (CLTV delta of Peer B)

Now after 144 blocks, the peer A will also force close on peer B just because peer B has not failed the HTLC backward.

This causes a cascade of FC in current mempool (and specially with peers with shorter CLTV delta).

There is a similar case with LDK -> lightningdevkit/rust-lightning#2275

Logs:
Peer B force closes on an offline peer after HTLC expiry.

/home/umbrel/lndlog/lnd.log.754.gz:2023-05-08 06:18:56.124 [INF] CNCT: ChannelArbitrator(0b228050fd8eeecf22073086a8885faf0c4d2bc02ad9480f23767368da411905:0): immediately failing htlc=3837313232636338363662613938653434353830343430613764383636646666 from remote commitment
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-08 06:18:56.204 [INF] CNCT: ChannelArbitrator(0b228050fd8eeecf22073086a8885faf0c4d2bc02ad9480f23767368da411905:0): force closing chan

The force close transaction is still in mempool.
144 blocks later peer A also force closed in a cascade

/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.021 [INF] CNCT: Unilateral close of ChannelPoint(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0) detected
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.048 [WRN] HSWC: ChannelLink(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0): remote peer has closed on-chain
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.050 [INF] HSWC: ChannelLink(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0): exited
/home/umbrel/lndlog/lnd.log.754.gz:2023-05-09 07:17:35.050 [INF] CNCT: ChannelArbitrator(4d8ef40c52865f007816a151510343bb89d8d36e2dd7e34edc4748a03027e087:0): remote party has closed channel out on-chain

The second force close would have been prevented if HTLC was failed backward by peer B after force close with Offline Peer.

Your environment

• version of lnd
  "version": "0.16.2-beta commit=v0.16.2-beta",

• which operating system (uname -a on *Nix)

Linux umbrel 5.10.17-v8+ #1421 SMP PREEMPT Thu May 27 14:01:37 BST 2021 aarch64 GNU/Linux

Steps to reproduce

See background.

Expected behaviour

When Peer B force closes on offline peer/forward peer, it should immediately fail the HTLC backward to prevent peer A force close.

Actual behaviour

Cascade of Force Close down the chain.

[ellemouton]

The thing is that Peer B wants to make sure that they can claim the timeout path before they fail back the htlc. Otherwise there is a chance that the offline peer comes back online just in time & then claims the success path. This will mean a loss of funds for peer B if they have already failed the htlc back to peer A

[TheBlueMatt]

Sure but if you're already out of time on the backwards path you run that risk anyway? We're thinking about this on the LDK end and I'm a bit torn but it does seem like the "common case" here is the cascading failure, not the attack, though it's possible that changes with package relay.

[BhaagBoseDK]

the offline peer should only be able to claim until the HTLC expiry. It should have no bearing on when the FC transaction is confirmed on-chain.

[ellemouton]

Sure but if you're already out of time on the backwards path you run that risk anyway?

Ah, that is a good point.

the offline peer should only be able to claim until the HTLC expiry.

unforch that is not possible to enforce with bitcoin Script. After the htlc expiry, the output becomes a free-for-all if the pre-image is known.

[BhaagBoseDK]

another point is this specific case of "missing HTLC in remote commitment". In the FC from peer B -> offline peer, the HTLC was not in the remote commitment (therefore there is no possibility for the offline peer to come back and claim it later). The FC transaction does not even have the HTLC.
8f58a419830c62f9e708b6c47b5541c044a19a1cdc64c4eb0c903311d6282fdd
In this case it could be safely failed the HTLC with peer A.

[ziggie1984]

another point is this specific case of "missing HTLC in remote commitment". In the FC from peer B -> offline peer, the HTLC was not in the remote commitment (therefore there is no possibility for the offline peer to come back and claim it later). The FC transaction does not even have the HTLC.
8f58a419830c62f9e708b6c47b5541c044a19a1cdc64c4eb0c903311d6282fdd
In this case it could be safely failed the HTLC with peer A.

Not really because you have to make sure that your Commitment (without the HTLC) is confirmed, because your Peer has a valid Commitment Transaction with the HTLC included (at least you think it has one, you cannot be sure it did not receive it because he was offline before), this means this HTLC could very much be confirmed when your Peer has the preimage and decides to go onchain.

[ziggie1984]

Ok I was having the same case with and Incoming and Outgoing HTLC being stuck because the outgoing HTLC was going onchain (and did not confirm until the incoming HTLC would run into the timeout). But luckily my incoming HTLC was failed back because of a positive side_effect of the interceptor. Basically the interceptor will fail all expiring incoming HTLCs which are close to expiry [13 blocks away]. (https://github.com/lightningnetwork/lnd/blob/master/htlcswitch/interceptable_switch.go#L470). Thats exactly what happened in my case, it canceled it exactly 13 blocks before timeout.

I think the important code part is here:

https://github.com/lightningnetwork/lnd/blob/master/htlcswitch/interceptable_switch.go#L293

here we cancel all incoming HTLCs although their outgoing counterpart is not resolved yet, at least we do not check whether there is an Outgoing HTLC on the downstream channel.

This failing of an incoming HTLC where the outgoing is still stuck is pretty new (9 months) and really hard to test in regtest mode (filling the mempool with unconfirmed TX), could you look into it @joostjager whether my analysis is correct :)

What I am saying is basically when you have an Interceptor running it will fail back Incoming HTLCs although their Outgoing counterpart is still not resolved yet. I think it's good, because otherwise your Peer will Force-Close on you anyways and you will lose the second channel.

[yyforyongyu]

Given the setup, Peer Alice -> Peer Bob -> Offline Peer Charlie, if Charlie is offline during the whole time, then yeah it's safe to cancel the HTLC, but you can't be sure. If Charlie comes online after the FC, there are two scenarios,

1. the HTLC hasn't timed out yet, Charlie can claim the HTLC via the preimage path, Bob will extract the preimage from mempool and settle the incoming HTLC with Alice.
2. the HTLC has timed out, and the sweeping tx is not yet confirmed, Charlie can still claim it by racing against Bob. In this case, Bob will extract the preimage and settle the HTLC with Alice.

This means Bob would not lose the HTLC if Charlie decides to come online and claim it for w/e reason. However, if Bob cancels the HTLC with Alice after the FC, he is at risk of losing it if Charlie decides to cheat.

So IMO canceling early is not a good choice. Instead, assuming this is an anchor channel, the most feasible way is to fee bump the force close tx.

[Crypt-iQ]

Think we can close this