Wallet TX count and time indexing

[braydonf]

This resolves CPU and memory exhaustion issues when requesting transaction history for a wallet with many transactions. Transactions are now also able to be queried by time based on median-time-past (MTP) based on chain data.

Closes: #559

• Indexes for confirmed txs
• Handle reorganizations
• Indexes for unconfirmed txs
• Complete API and exposed methods and endpoints
• Cleanup, refactoring and dependencies
• Testing/review, documentation, upgrading

See TODO.md for a full list.

[bucko13]

Looking great! Excited to see the progress on this.

[braydonf]

Closing this pull request for now, as I haven't worked on this in quite some time. This solves the issue with many transactions within a wallet, and I've added this branch at https://github.com/bcoin-org/bcoin/tree/wallet-pagination and will leave it as it is currently until this topic is visited again. There is also the topic of the ability to recover the wallet from a seed that is important.

[nodech]

Bug related to unconfirmed index

Bug is related to how unconfirmed count recovery happens. It may rewrite the existing last transaction details. Here's is the idea how it could happen:

Situation 1:
- insert unconfirmed tx1. (increments layout.z.UNCONFIRMED)
- confirm tx1. (this store count in layout.x) / decrements layout.z.UNCONFIRMED
- insert new unconfirmed tx2. unconfirmed count for this will be the SAME as tx1.
- reorganization undos tx1. - layout.x - will be the same.

this can happen on scale as well:
- Add N unconfirmed txs.
- Confirm N txs in a block.
- Add new N txs.
- Reorganize so they are unmined.

Solution

• Solution 1: never ever decrement the unconfirmed count. This needs new entry in the txdb, that will track the max unconfirmed index. This could avoid all the issues related to recovery on disconnect/unconfirm. Downside is that it may hasten the overflow of U32 count.index. For huge wallets, this may become an issue at some point.

  • NOTE: When tx is first seen directly in block (insert \w block), we need to increment unconfirmed count as well, to make sure it has reserved index in the unconfirmed layout.

• Solution 2: Never recover and put them always on top. This can be more flexible, does not suffer from the overflow issue. Increment/decrement always happens on top, behaves like a stack and we can drop layout.x altogether. Downside is that it will violate the cronology and old confirmed but disconnected txs, will become new txs. To not violate the time index, we could also drop layout.e that recovers time and treat them as new txs.

Test case

Apply to test/wallet-test.js

  describe('Pagination indexes', function() {
    let wdb;

    before(async () => {
      wdb = new WalletDB({ memory: true });
      await wdb.open();
    });

    after(async () => {
      await wdb.close();
    });

    it('should reindex newly unconfirmed txs after disconnect', async () => {
      const N = 2;

      const toConfirm = [];
      for (let i = 0; i < N; i++) {
        const address = await wdb.primary.receiveAddress();
        const mtx = new MTX();
        mtx.addInput(dummyInput());
        mtx.addOutput(address, 1);

        const tx = mtx.toTX();
        toConfirm.push(tx);
        await wdb.addTX(tx);
      }

      {
        const unconfirmed = await wdb.primary.listUnconfirmed(0, {
          limit: 100,
          reverse: false
        });

        assert.strictEqual(unconfirmed.length, N);
      }

      const entry = nextBlock(wdb);
      await wdb.addBlock(entry, toConfirm);

      {
        const unconfirmed = await wdb.primary.listUnconfirmed(0, {
          limit: 100,
          reverse: false
        });

        assert.strictEqual(unconfirmed.length, 0);
      }

      for (let i = 0; i < N; i++) {
        const address = await wdb.primary.receiveAddress();
        const mtx = new MTX();
        mtx.addInput(dummyInput());
        mtx.addOutput(address, 1);

        const tx = mtx.toTX();
        toConfirm.push(tx);
        await wdb.addTX(tx);
      }

      {
        const unconfirmed = await wdb.primary.listUnconfirmed(0, {
          limit: 100,
          reverse: false
        });

        const all = await wdb.primary.listHistory(0, {
          limit: 100,
          reverse: false
        })

        assert.strictEqual(unconfirmed.length, N);
        assert.strictEqual(all.length, N * 2);
      }

      console.log(await wdb.dump());

      await wdb.removeBlock(entry);

      {
        const unconfirmed = await wdb.primary.listUnconfirmed(0, {
          limit: 100,
          reverse: false
        });

        const all = await wdb.primary.listHistory(0, {
          limit: 100,
          reverse: false
        })

        console.log(await wdb.dump());
        assert.strictEqual(unconfirmed.length, N * 2);
        assert.strictEqual(all.length, N * 2);
      }
    });
  });

[nodech]

typo here. Should be x: bdb.key('x') instead of bdb.key('u')