Test compatibility with joinmarket

[DanGould]

• send from JoinMarket to payjoin-cli. JoinMarket may need to be able to trust self-signed certs or otherwise a cert from e.g. letsencrypt may need to be obtained. This is a priority to smoke test compatibility for production integration @ Bull Bitcoin
• receive tested from JoinMarket. JoinMarket receives payjoin via Tor, so SOCKS Tor support would need to be added to payjoin-cli

payjoin/payjoin.org#8

[spacebear21]

I've tested sending from JoinMarket to payjoin-cli and found the bug described below.

Setup notes

JoinMarket doesn't trust self-signed certificates, so I had to obtain a legit certificate from letsencrypt.org as you described.
I launched an EC2 instance with nginx acting as a proxy for payjoin-cli, running on signet. I was able to test this setup by sending a successful payjoin transaction from payjoin-cli on my local machine, to the payjoin receiver listening behind the nginx proxy.

Receive error when sending from JM

However, sending from joinmarket using the same setup results in a FinalizeAndExtractPSBT error on the receiver side.

RUST_LOG=debug cargo run -- receive 10000
    Finished dev [unoptimized + debuginfo] target(s) in 0.14s
     Running `/home/ec2-user/rust-payjoin/target/debug/payjoin-cli receive 10000`
[2024-04-18T22:43:46Z DEBUG payjoin_cli::app::config] App config: AppConfig { bitcoind_rpchost: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(38332), path: "/wallet/pj_wallet", query: None, fragment: None }, bitcoind_cookie: None, bitcoind_rpcuser: "***", bitcoind_rpcpass: "***", pj_host: 0.0.0.0:3000, pj_endpoint: Url { scheme: "https", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(3000), path: "/", query: None, fragment: None }, sub_only: false }
[2024-04-18T22:43:46Z DEBUG bitcoincore_rpc] JSON-RPC request: getnewaddress [null,null]
Listening at 0.0.0.0:3000. Configured to accept payjoin at BIP 21 Payjoin Uri:
bitcoin:tb1qn9dt8hmcg87cvz2rpmclskvp4lv5xdfj44qh5q?amount=0.0001&pj=https://localhost:3000/&pjos=0
[2024-04-18T22:45:54Z DEBUG payjoin_cli::app::v1] Received request: Request { method: POST, uri: /?v=1&disableoutputsubstitution=true&additionalfeeoutputindex=0&maxadditionalfeecontribution=95&minfeerate=1.1, version: HTTP/1.0, headers: {"host": "pj.spacebear.dev", "x-forwarded-proto": "https", "connection": "close", "content-length": "360", "content-type": "text/plain"}, body: Body(Streaming) }
[2024-04-18T22:45:54Z DEBUG payjoin::receive] Received original psbt: PartiallySignedTransaction { unsigned_tx: Transaction { version: 2, lock_time: Blocks(Height(191867)), input: [TxIn { previous_output: OutPoint { txid: 0xc6ffb1bb73040f88b520cd7869e28b2cbeebc320cc42f67a84e3107306a9ea3b, vout: 1 }, script_sig: Script(), sequence: Sequence(4294967293), witness: Witness { content: [], witness_elements: 0, indices_start: 0 } }], output: [TxOut { value: 949217, script_pubkey: Script(OP_0 OP_PUSHBYTES_20 35bef6d37390ea20c43ba951cc2a30831166399b) }, TxOut { value: 10000, script_pubkey: Script(OP_0 OP_PUSHBYTES_20 995ab3df7841fd8609430ef1f85981afd9433532) }] }, version: 0, xpub: {}, proprietary: {}, unknown: {}, inputs: [Input { non_witness_utxo: None, witness_utxo: Some(TxOut { value: 959382, script_pubkey: Script(OP_0 OP_PUSHBYTES_20 4d20ca992bee15e856dfefa1be0b1ad2a0c69fc9) }), partial_sigs: {}, sighash_type: None, redeem_script: None, witness_script: None, bip32_derivation: {}, final_script_sig: None, final_script_witness: Some(Witness { content: [71, 48, 68, 2, 32, 110, 221, 75, 43, 61, 10, 149, 172, 84, 232, 107, 84, 126, 26, 61, 206, 44, 218, 21, 171, 210, 9, 24, 3, 3, 153, 13, 99, 28, 206, 92, 30, 2, 32, 35, 237, 171, 254, 193, 89, 175, 223, 188, 120, 4, 209, 22, 202, 10, 145, 197, 227, 255, 25, 70, 229, 76, 198, 51, 21, 227, 222, 37, 174, 113, 209, 1, 33, 3, 133, 51, 208, 148, 147, 127, 254, 66, 145, 179, 7, 93, 245, 123, 11, 154, 188, 30, 55, 35, 40, 97, 250, 126, 181, 20, 231, 187, 3, 108, 118, 3, 0, 0, 0, 0, 72, 0, 0, 0], witness_elements: 2, indices_start: 106 }), ripemd160_preimages: {}, sha256_preimages: {}, hash160_preimages: {}, hash256_preimages: {}, tap_key_sig: None, tap_script_sigs: {}, tap_scripts: {}, tap_key_origins: {}, tap_internal_key: None, tap_merkle_root: None, proprietary: {}, unknown: {} }], outputs: [Output { redeem_script: None, witness_script: None, bip32_derivation: {}, tap_internal_key: None, tap_tree: None, tap_key_origins: {}, proprietary: {}, unknown: {} }, Output { redeem_script: None, witness_script: None, bip32_derivation: {}, tap_internal_key: None, tap_tree: None, tap_key_origins: {}, proprietary: {}, unknown: {} }] }
[2024-04-18T22:45:54Z DEBUG payjoin::receive::optional_parameters] parsed optional parameters: Params { disable_output_substitution: true, additional_fee_contribution: Some((Amount(0.00000095 BTC), 0)), min_feerate: FeeRate(275) }
[2024-04-18T22:45:54Z DEBUG payjoin::receive] Received request with params: Params { disable_output_substitution: true, additional_fee_contribution: Some((Amount(0.00000095 BTC), 0)), min_feerate: FeeRate(275) }
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: getblockchaininfo []
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: getnetworkinfo []
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: testmempoolaccept [["020000000001013beaa9067310e3847af642cc20c3ebbe2c8be26978cd20b5880f0473bbb1ffc60100000000fdffffff02e17b0e000000000016001435bef6d37390ea20c43ba951cc2a30831166399b1027000000000000160014995ab3df7841fd8609430ef1f85981afd94335320247304402206edd4b2b3d0a95ac54e86b547e1a3dce2cda15abd209180303990d631cce5c1e022023edabfec159afdfbc7804d116ca0a91c5e3ff1946e54cc63315e3de25ae71d10121038533d094937ffe4291b3075df57b0b9abc1e37232861fa7eb514e7bb036c76037bed0200"]]
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: getaddressinfo ["tb1qf5sv4xftac27s4kla7smuzc662svd87fgs02yu"]
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: getaddressinfo ["tb1qxkl0d5mnjr4zp3pm49guc23ssvgkvwvm429wyj"]
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: getaddressinfo ["tb1qn9dt8hmcg87cvz2rpmclskvp4lv5xdfj44qh5q"]
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: listunspent []
[2024-04-18T22:45:54Z DEBUG payjoin_cli::app] selected utxo: ListUnspentResultEntry {
        txid: 0x76262ad10fbbc4692239d74e261ebf9ecb19ddeaea10460e38cf0e1ab8f1e0ad,
        vout: 0,
        address: Some(
            Address<NetworkUnchecked>(TB1QKRMXNUPZNEVSNPUVY9VAN6TGA9UHZGQPJ39FNG),
        ),
        label: Some(
            "",
        ),
        redeem_script: None,
        witness_script: None,
        script_pub_key: Script(OP_0 OP_PUSHBYTES_20 b0f669f0229e5909878c2159d9e968e979712001),
        amount: Amount(0.00899792 BTC),
        confirmations: 1,
        spendable: true,
        solvable: true,
        descriptor: Some(
            "wpkh([90a5aa64/84h/1h/0h/0/20]02ac17bc19cf73e28f1eb29343f0a30e333fdac505c0502e4886b990baa6c69ba7)#gljrxr7w",
        ),
        safe: true,
    }
[2024-04-18T22:45:54Z DEBUG bitcoincore_rpc] JSON-RPC request: getnewaddress [null,null]
[2024-04-18T22:45:55Z DEBUG payjoin::receive] min_feerate: FeeRate(275)
[2024-04-18T22:45:55Z DEBUG bitcoincore_rpc] JSON-RPC request: walletprocesspsbt ["cHNidP8BAJoCAAAAAq3g8bgaDs84DkYQ6urdGcuevx4mTtc5ImnEuw/RKiZ2AAAAAAD9////O+qpBnMQ44R69kLMIMPrviyL4ml4zSC1iA8Ec7ux/8YBAAAAAP3///8ClnsOAAAAAAAWABQ1vvbTc5DqIMQ7qVHMKjCDEWY5m+DhDQAAAAAAFgAUWIMWf8n1WEfQh3v8yh+J7jCs/a577QIAAAEBH9C6DQAAAAAAFgAUsPZp8CKeWQmHjCFZ2elo6XlxIAEAAQEflqMOAAAAAAAWABRNIMqZK+4V6Fbf76G+CxrSoMafyQEIawJHMEQCIG7dSys9CpWsVOhrVH4aPc4s2hWr0gkYAwOZDWMczlweAiAj7av+wVmv37x4BNEWygqRxeP/GUblTMYzFePeJa5x0QEhA4Uz0JSTf/5CkbMHXfV7C5q8HjcjKGH6frUU57sDbHYDAAAA",true,"ALL",false]
[2024-04-18T22:45:55Z DEBUG bitcoincore_rpc] JSON-RPC error for walletprocesspsbt: RpcError { code: -1, message: "Internal bug detected: FinalizeAndExtractPSBT(psbtx, mtx)\nwallet/rpc/spend.cpp:1619 (operator())\nBitcoin Core v26.0.0\nPlease report this issue here: https://github.com/bitcoin/bitcoin/issues\n", data: None }
[2024-04-18T22:45:55Z ERROR payjoin_cli::app::v1] Error handling request: Internal Server Error: JSON-RPC error: RPC error response: RpcError { code: -1, message: "Internal bug detected: FinalizeAndExtractPSBT(psbtx, mtx)\nwallet/rpc/spend.cpp:1619 (operator())\nBitcoin Core v26.0.0\nPlease report this issue here: https://github.com/bitcoin/bitcoin/issues\n", data: None }

[DanGould]

Awesome seeing this compatibility put to the test.

This is the decoded psbt that caused the error which may help diagnose the problem

bitcoin-cli decodepsbt "cHNidP8BAJoCAAAAAq3g8bgaDs84DkYQ6urdGcuevx4mTtc5ImnEuw/RKiZ2AAAAAAD9////O+qpBnMQ44R69kLMIMPrviyL4ml4zSC1iA8Ec7ux/8YBAAAAAP3///8ClnsOAAAAAAAWABQ1vvbTc5DqIMQ7qVHMKjCDEWY5m+DhDQAAAAAAFgAUWIMWf8n1WEfQh3v8yh+J7jCs/a577QIAAAEBH9C6DQAAAAAAFgAUsPZp8CKeWQmHjCFZ2elo6XlxIAEAAQEflqMOAAAAAAAWABRNIMqZK+4V6Fbf76G+CxrSoMafyQEIawJHMEQCIG7dSys9CpWsVOhrVH4aPc4s2hWr0gkYAwOZDWMczlweAiAj7av+wVmv37x4BNEWygqRxeP/GUblTMYzFePeJa5x0QEhA4Uz0JSTf/5CkbMHXfV7C5q8HjcjKGH6frUU57sDbHYDAAAA"
{
  "tx": {
    "txid": "d839f0dbd2f27817d0be1920b346001ca110c591b7f7928fc2d349dece95a0d7",
    "hash": "d839f0dbd2f27817d0be1920b346001ca110c591b7f7928fc2d349dece95a0d7",
    "version": 2,
    "size": 154,
    "vsize": 154,
    "weight": 616,
    "locktime": 191867,
    "vin": [
      {
        "txid": "76262ad10fbbc4692239d74e261ebf9ecb19ddeaea10460e38cf0e1ab8f1e0ad",
        "vout": 0,
        "scriptSig": {
          "asm": "",
          "hex": ""
        },
        "sequence": 4294967293
      },
      {
        "txid": "c6ffb1bb73040f88b520cd7869e28b2cbeebc320cc42f67a84e3107306a9ea3b",
        "vout": 1,
        "scriptSig": {
          "asm": "",
          "hex": ""
        },
        "sequence": 4294967293
      }
    ],
    "vout": [
      {
        "value": 0.00949142,
        "n": 0,
        "scriptPubKey": {
          "asm": "0 35bef6d37390ea20c43ba951cc2a30831166399b",
          "desc": "addr(bc1qxkl0d5mnjr4zp3pm49guc23ssvgkvwvmlv7alp)#gjwwrf32",
          "hex": "001435bef6d37390ea20c43ba951cc2a30831166399b",
          "address": "bc1qxkl0d5mnjr4zp3pm49guc23ssvgkvwvmlv7alp",
          "type": "witness_v0_keyhash"
        }
      },
      {
        "value": 0.00909792,
        "n": 1,
        "scriptPubKey": {
          "asm": "0 5883167fc9f55847d0877bfcca1f89ee30acfdae",
          "desc": "addr(bc1qtzp3vl7f74vy05y8007v58ufacc2eldwgkdk6y)#kljvka6k",
          "hex": "00145883167fc9f55847d0877bfcca1f89ee30acfdae",
          "address": "bc1qtzp3vl7f74vy05y8007v58ufacc2eldwgkdk6y",
          "type": "witness_v0_keyhash"
        }
      }
    ]
  },
  "global_xpubs": [
  ],
  "psbt_version": 0,
  "proprietary": [
  ],
  "unknown": {
  },
  "inputs": [
    {
      "witness_utxo": {
        "amount": 0.00899792,
        "scriptPubKey": {
          "asm": "0 b0f669f0229e5909878c2159d9e968e979712001",
          "desc": "addr(bc1qkrmxnupznevsnpuvy9van6tga9uhzgqpch76gm)#z2welvn9",
          "hex": "0014b0f669f0229e5909878c2159d9e968e979712001",
          "address": "bc1qkrmxnupznevsnpuvy9van6tga9uhzgqpch76gm",
          "type": "witness_v0_keyhash"
        }
      }
    },
    {
      "witness_utxo": {
        "amount": 0.00959382,
        "scriptPubKey": {
          "asm": "0 4d20ca992bee15e856dfefa1be0b1ad2a0c69fc9",
          "desc": "addr(bc1qf5sv4xftac27s4kla7smuzc662svd87fzk5el0)#4y3q79p9",
          "hex": "00144d20ca992bee15e856dfefa1be0b1ad2a0c69fc9",
          "address": "bc1qf5sv4xftac27s4kla7smuzc662svd87fzk5el0",
          "type": "witness_v0_keyhash"
        }
      },
      "final_scriptwitness": [
        "304402206edd4b2b3d0a95ac54e86b547e1a3dce2cda15abd209180303990d631cce5c1e022023edabfec159afdfbc7804d116ca0a91c5e3ff1946e54cc63315e3de25ae71d101",
        "038533d094937ffe4291b3075df57b0b9abc1e37232861fa7eb514e7bb036c7603"
      ]
    }
  ],
  "outputs": [
    {
    },
    {
    }
  ],
  "fee": 0.00000240
}

[spacebear21]

This is what i think is happening:

1. Joinmarket sends the PSBT with its own input signed.
   • This PSBT input only includes a witness_utxo (no non_witness_utxo), which is legal/encouraged by BIP 174.
   • The final_scriptwitness for that input has a signature which commits to all outputs in the transaction.
2. The payjoin-cli receiver runs some checks, then substitutes its own output address (scriptPubKey) with a new one.
   • Here, the receiver appears to disregard the disableoutputsubstitution=true flag in the POST request from JM sender. It substitutes its output regardless.
3. The payjoin-cli receiver attempts to finalize the psbt via walletprocesspsbt RPC call, but the signature in final_scriptwitness from (1) is no longer valid due to the output substitution.
   • Bitcoin Core fails to FinalizeAndExtractPSBT and throws a CHECK_NONFATAL failure (It should probably handle this scenario gracefull instead...)
   • iiuc the PSBT doesn't need to be finalized at this stage (payjoin proposal). I can get walletprocesspsbt to work on this PSBT manually via bitcoin-cli by setting the finalize flag to false. Unfortunately it looks like the rust bitcoincore-rpc library doesn't support finalize yet.

The reason it works with the payjoin-cli sender is because the sender populates both witness_utxo and non_witness_utxo for the input(s) it contributes, which is also allowed by BIP 174. Bitcoin Core sees the non_witness_utxo, so it doesn't require a valid witness sig to finalize the PSBT.

TL;DR:

• payjoin-cli receiver should respect disableoutputsubstitution. See: payjoin-cli should respect output substitution #238
• payjoin-cli receiver doesn't need to finalize PSBT proposals before sending them back to the sender.