'withdraw all' fails when 'utxos' specified; leaves UTxOs unavailable

[whitslack]

Issue and Steps to Reproduce

$ newaddr=$(lightning-cli newaddr | jq -r .bech32)

$ lightning-cli withdraw "${newaddr}" all 3333perkb 0 '["(my txid):(vout)"]'
{
   "code": 301,
   "message": "Cannot afford transaction with 138sat sats of fees"
}

Attempting the same withdrawal a second time yields:

{
   "code": -32602,
   "message": "No matching utxo was found from the wallet. You can get a list of the wallet utxos with the `listfunds` RPC call."
}

Indeed, the specified <txid>:<vout> is no longer returned in lightning-cli listfunds. The UTxO does appear again after a dev-rescan-outputs, but obviously this is risky and shouldn't be needed.

So there are really two problems here:

1. C-Lightning should not be marking UTxOs as spent if it didn't actually spend them.
2. C-Lightning apparently doesn't understand how to spend all of a specified set of UTxOs while deducting enough from their total value to satisfy the specified fee rate.

getinfo output

$ lightning-cli getinfo | jq '{binding, version, blockheight, network, "lightning-dir"}'
{
  "binding": [
    {
      "type": "ipv6",
      "address": "::",
      "port": 9735
    },
    {
      "type": "ipv4",
      "address": "0.0.0.0",
      "port": 9735
    }
  ],
  "version": "0.8.1",
  "blockheight": 621608,
  "network": "bitcoin",
  "lightning-dir": "/var/lib/lightning/bitcoin"
}

[whitslack]

Problem 1 also occurs when attempting to fundchannel with a specified UTxO but the channel funding fails because the auto-connection to the target node failed. Specifically, C-Lightning keeps the UTxO reserved, even though no attempt was actually made to spend it.

[darosior]

I experienced it too with fundchannel. And even funnier things happen when you try to fundchannel_cancel the peer.. Giving a look to withdraw now.

You should not specify all, but we shouldn't crash.

[whitslack]

You should not specify all, but we shouldn't crash.

@darosior: Okay, then what is the recommended way to withdraw (or fund a channel with) exactly a set of UTxOs without producing a change output? The all keyword seems like the most logical choice to me. I expect it to take all coins available in the specified set of UTxOs and send them, minus a specified mining fee.

[darosior]

Hmm all specifies to spend all the available funds from the wallet.
But you are right that it would be nice to be able to condition it to the specified utxos..

EDIT: to answer the question clearly, you should set the amount explicitly if you provide a list of utxos.

[whitslack]

to answer the question clearly, you should set the amount explicitly if you provide a list of utxos.

But I don't know what the fee will be. If I specify the whole sum of the UTxOs, then there won't be anything left for the fee, and if I specify too little a sum, then I'll inadvertently generate a change output, which is what I'm trying to avoid.

[darosior]

Yes, that's the well-known problem of fees computation, that's why it's better automated :-)
Though note that we won't create a dust output, which leaves some room for approximation.

[darosior]

@whitslack ok I tried to reproduce and I think I understood the root cause : were you using unconfirmed utxos ?

EDIT: btw I was badly wrong, all is handled along with utxos

[whitslack]

were you using unconfirmed utxos ?

@darosior: No. I was specifying exactly one confirmed UTxO, and I was specifying an explicit feerate using the perkb suffix. I have some tiny, confirmed UTxOs that I am willing to try withdrawing from my C-Lightning wallet if you'd like me to do any tests. Everytime I've tried before, using the all keyword, I get the Cannot afford transaction with ___sat sats of fees error. (Side note: That error message says "sat" twice.)

[darosior]

Weird.. Then I cannot reproduce on current master. Cf #3593 for the unconfirmed case..

[whitslack]

Maybe it's already been fixed since the 0.8.1 release. I'll retest after the next release and update here with my findings.

[niftynei]

@whitslack has this been fixed?

[whitslack]

@niftynei: I don't know. There hasn't been a new release since 0.8.1. I'll update this issue again after there's been a new release and I've retested.

[niftynei]

Oh I see, apologies I didn't quite grok that you were waiting til the next release to test :)

[ZmnSCPxj]

We already handle this correctly, as these tests show, and these tests are in released 0.8.2.1:

lightning/tests/test_wallet.py

Line 172 in 7a7b154

l1.rpc.withdraw(waddr, "all", minconf=0, utxos=uutxos)

lightning/tests/test_connection.py

Line 872 in 7a7b154

l1.rpc.fundchannel(l2.info["id"], "all", utxos=utxos[0:2])

@whitslack please confirm if ok to close.