Wallet TX count and time indexing

CHANGELOG.md

@@ -45,8 +45,19 @@ before upgrading.
 
 ### Wallet API changes
 
+The wallet has been updated to handle a large number of transactions, there
+are several new API methods that have been added and modified to better
+support transaction queries. This also includes the ability to query
+for transaction history by median-time-past (MTP). It is necessary to
+perform a wallet rescan to update the wallet indexes.
+
 #### HTTP
 
+These endpoints have been modified:
+
+- `GET /wallet/:id/tx/history` - The params are now `time`, `after`,
+  `limit`, and `reverse`.
+- `GET /wallet/:id/tx/unconfirmed` - The params are the same as above.
 - `PUT /wallet/:id` Creating a watch-only wallet now requires an `accountKey`
   argument. This is to prevent bcoin from generating keys and addresses the
   user can not spend from.
@@ -58,19 +69,81 @@ before upgrading.
   - Exposes `blocks`, which can will be used if there is no `rate` option.
   - Exposes `sort` (Default `true`), that can be used to disable BIP69 sorting.
 
+These endpoints have been deprecated:
+
+- `GET /wallet/:id/tx/range` - Instead use the `time` param for the history and
+  unconfirmed endpoints.
+- `GET /wallet/:id/tx/last` - Instead use `reverse` param for the history and
+  unconfirmed endpoints.
+
+##### Examples
+
+```
+GET /wallet/:id/tx/history?after=<txid>&limit=50&reverse=false
+GET /wallet/:id/tx/history?after=<txid>&limit=50&reverse=true
+```
+By using `after=<txid>` we can anchor pages so that results will not shift
+when new blocks and transactions arrive. With `reverse=true` we can change
+the order the transactions are returned as _latest to genesis_. The
+`limit=<number>` specifies the maximum number of transactions to return
+in the result.
+
+```
+GET /wallet/:id/tx/history?time=<median-time-past>&limit=50&reverse=false
+GET /wallet/:id/tx/history?time=<median-time-past>&limit=50&reverse=true
+```
+The param `time` is in epoch seconds and indexed based on median-time-past
+(MTP) and `date` is ISO 8601 format. Because multiple transactions can share
+the same time, this can function as an initial query, and then switch to the
+above `after` format for the following pages.
+
+```
+GET /wallet/:id/tx/unconfirmed?after=<txid>&limit=50&reverse=false
+GET /wallet/:id/tx/unconfirmed?after=<txid>&limit=50&reverse=true
+GET /wallet/:id/tx/unconfirmed?time=<time-received>&limit=50&reverse=false
+```
+The same will apply to unconfirmed transactions. The `time` is in epoch
+seconds and indexed based on when the transaction was added to the wallet.
+
 #### RPC
 
-- Bug fix addresses for the `getnewaddress` command with various networks.
-- Deprecate the `ismine` and `iswatchonly` fields from the `validateaddress`
-  command and add `isscript`, `iswitness`, `ischange`, `witness_version`
-  and `witness_program` to partially match the v0.18.0 Bitcoin Core release
+The following methods have bug fixes:
+- `getnewaddress` Now supports various networks.
+
+The following have property deprecations and additions:
+- `validateaddress` - Deprecate the `ismine` and `iswatchonly` properties,
+  use the new command `getaddressinfo` instead. The command now has
+  `isscript`, `iswitness`, `ischange`, `witness_version` and
+  `witness_program` to partially match the v0.18.0 Bitcoin Core release
   (26a2000b0177fd2668b7d82e5aa52829cf2bfdf6)
-- Add wallet RPC `getaddressinfo` to return `ismine` and `iswatchonly`
-  with the correct values instead of their previous values which were
-  hardcoded to false. Also returns `address`, `scriptPubKey`, `isscript`,
-  `iswitness`, `witness_version` and `witness_program`.
+
+The following new methods have been added:
+
+- `listhistory` - List history with a limit and in reverse order.
+- `listhistoryafter` - List history after a txid _(subsequent pages)_.
+- `listhistorybytime` - List history by giving a timestamp in epoch seconds
+  _(block median time past)_.
+- `listunconfirmed` - List unconfirmed transactions with a limit and in
+  reverse order.
+- `listunconfirmedafter` - List unconfirmed transactions after a txid
+  _(subsequent pages)_.
+- `listunconfirmedbytime` - List unconfirmed transactions by time they
+  where added.
+- `getaddressinfo` - Returns `ismine` and `iswatchonly` with the correct
+  values instead of their previous values from `validateaddress` which
+  were hardcoded to false.  Also returns `address`, `scriptPubKey`,
+  `isscript`, `iswitness`, `witness_version` and `witness_program`.
   (a28ffa272a3c4d90d0273d9aa223a23becc08e0e)
 
+The following methods have been deprecated:
+
+- `listtransactions` - Use `listhistory` and the related methods and the
+  `after` argument for results that do not shift when new blocks arrive.
+- `importprunedfunds` - Would cause a shift in transaction counts, and better
+  handled by adding and removing blocks at a time via resetting and
+  rescanning pruned and spv nodes.
+- `removeprunedfunds` - For similar reasons as `importprunedfunds`.
+
 ### Node API changes
 
 #### HTTP
@@ -125,6 +198,12 @@ that have many transactions.
 - The database location for indexes can be configured via the
   `--index-prefix` option. Default locations are `prefix` + `/index`
   (e.g. `~/.bcoin/testnet/index/tx` and `~/.bcoin/testnet/index/addr`).
+- To resolve CPU and memory exhaustion issues for wallets with many
+  transactions, an new option was introduced to for the maximum number
+  of transactions returned in a page of results for wallets. The maximum
+  number can be changed based on individual resource limits with the
+  configuration option `wallet-max-txs`, it's unlikely that this value
+  should need to be changed.
 
 ### Script changes
 

lib/wallet/client.js

@@ -63,6 +63,18 @@ class WalletClient extends NodeClient {
     return parseEntry(await super.getEntry(block));
   }
 
+  async getMedianTime(blockhash) {
+    if (Buffer.isBuffer(blockhash))
+      blockhash = util.revHex(blockhash);
+
+    const blockheader = await super.execute('getblockheader', [blockhash]);
+
+    if (!blockheader)
+      return null;
+
+    return blockheader.mediantime;
+  }
+
   async send(tx) {
     return super.send(tx.toRaw());
   }

lib/wallet/common.js

@@ -50,18 +50,6 @@ common.isName = function isName(key) {
   return key.length >= 1 && key.length <= 40;
 };
 
-/**
- * Sort an array of transactions by time.
- * @param {TX[]} txs
- * @returns {TX[]}
- */
-
-common.sortTX = function sortTX(txs) {
-  return txs.sort((a, b) => {
-    return a.mtime - b.mtime;
-  });
-};
-
 /**
  * Sort an array of coins by height.
  * @param {Coin[]} txs

lib/wallet/http.js

@@ -767,9 +767,26 @@ class HTTP extends Server {
     this.get('/wallet/:id/tx/history', async (req, res) => {
       const valid = Validator.fromRequest(req);
       const acct = valid.str('account');
-      const txs = await req.wallet.getHistory(acct);
-
-      common.sortTX(txs);
+      const reverse = valid.bool('reverse', false);
+      const limit = valid.u32('limit', 10);
+      const after = valid.brhash('after');
+      const time = valid.u32('time');
+
+      enforce(limit <= this.options.maxTxs,
+              `Limit above max of ${this.options.maxTxs}.`);
+
+      let txs = [];
+      const opts = {limit, reverse};
+
+      if (after) {
+        opts.hash = after;
+        txs = await req.wallet.listHistoryAfter(acct, opts);
+      } else if (time) {
+        opts.time = time;
+        txs = await req.wallet.listHistoryByTime(acct, opts);
+      } else {
+        txs = await req.wallet.listHistory(acct, opts);
+      }
 
       const details = await req.wallet.toDetails(txs);
 
@@ -785,47 +802,27 @@ class HTTP extends Server {
     this.get('/wallet/:id/tx/unconfirmed', async (req, res) => {
       const valid = Validator.fromRequest(req);
       const acct = valid.str('account');
-      const txs = await req.wallet.getPending(acct);
-
-      common.sortTX(txs);
-
-      const details = await req.wallet.toDetails(txs);
-      const result = [];
-
-      for (const item of details)
-        result.push(item.toJSON(this.network, this.wdb.height));
-
-      res.json(200, result);
-    });
-
-    // Wallet TXs within time range
-    this.get('/wallet/:id/tx/range', async (req, res) => {
-      const valid = Validator.fromRequest(req);
-      const acct = valid.str('account');
-
-      const options = {
-        start: valid.u32('start'),
-        end: valid.u32('end'),
-        limit: valid.u32('limit'),
-        reverse: valid.bool('reverse')
-      };
-
-      const txs = await req.wallet.getRange(acct, options);
-      const details = await req.wallet.toDetails(txs);
-      const result = [];
-
-      for (const item of details)
-        result.push(item.toJSON(this.network, this.wdb.height));
-
-      res.json(200, result);
-    });
+      const reverse = valid.bool('reverse', false);
+      const limit = valid.u32('limit', 10);
+      const after = valid.brhash('after');
+      const time = valid.u32('time');
+
+      enforce(limit <= this.options.maxTxs,
+              `Limit above max of ${this.options.maxTxs}.`);
+
+      let txs = [];
+      const opts = {limit, reverse};
+
+      if (after) {
+        opts.hash = after;
+        txs = await req.wallet.listUnconfirmedAfter(acct, opts);
+      } else if (time) {
+        opts.time = time;
+        txs = await req.wallet.listUnconfirmedByTime(acct, opts);
+      } else {
+        txs = await req.wallet.listUnconfirmed(acct, opts);
+      }
 
-    // Last Wallet TXs
-    this.get('/wallet/:id/tx/last', async (req, res) => {
-      const valid = Validator.fromRequest(req);
-      const acct = valid.str('account');
-      const limit = valid.u32('limit');
-      const txs = await req.wallet.getLast(acct, limit);
       const details = await req.wallet.toDetails(txs);
       const result = [];
 
@@ -1051,6 +1048,7 @@ class HTTPOptions {
     this.noAuth = false;
     this.cors = false;
     this.walletAuth = false;
+    this.maxTxs = 100;
 
     this.prefix = null;
     this.host = '127.0.0.1';
@@ -1158,6 +1156,11 @@ class HTTPOptions {
       this.certFile = options.certFile;
     }
 
+    if (options.maxTxs != null) {
+      assert(Number.isSafeInteger(options.maxTxs));
+      this.maxTxs = options.maxTxs;
+    }
+
     // Allow no-auth implicitly
     // if we're listening locally.
     if (!options.apiKey) {

lib/wallet/layout.js

@@ -24,6 +24,7 @@ const bdb = require('bdb');
  *  i[wid][name] -> account index
  *  n[wid][index] -> account name
  *  h[height] -> recent block hash
+ *  m[hash] -> block median time past
  *  b[height] -> block->wid map
  *  o[hash][index] -> outpoint->wid map
  *  T[hash] -> tx->wid map
@@ -45,6 +46,7 @@ exports.wdb = {
   i: bdb.key('i', ['uint32', 'ascii']),
   n: bdb.key('n', ['uint32', 'uint32']),
   h: bdb.key('h', ['uint32']),
+  m: bdb.key('m', ['hash256']),
   b: bdb.key('b', ['uint32']),
   o: bdb.key('o', ['hash256', 'uint32']),
   T: bdb.key('T', ['hash256']),
@@ -53,38 +55,76 @@ exports.wdb = {
 
 /*
  * TXDB Database Layout:
+ *   Balance
+ *   -------
  *   R -> wallet balance
  *   r[account] -> account balance
+ *
+ *   Transactions
+ *   ------------
  *   t[hash] -> extended tx
+ *   T[account][hash] -> dummy (tx by account)
+ *   z[height][index] -> dummy (tx by count)
+ *   Z[account][height][index]-> dummy (tx by count + account)
+ *   y[hash] -> count (count for tx)
+ *   x[hash] -> undo count (undo count for tx)
+ *
+ *   Coins
+ *   -----
  *   c[hash][index] -> coin
+ *   C[account][hash][index] -> dummy (coin by account)
  *   d[hash][index] -> undo coin
  *   s[hash][index] -> spent by hash
- *   p[hash] -> dummy (pending flag)
- *   m[time][hash] -> dummy (tx by time)
+ *
+ *   Confirmed
+ *   ---------
+ *   g[time][height][index][hash] -> dummy (tx by time)
+ *   G[account][time][height][index][hash] -> dummy (tx by time + account)
  *   h[height][hash] -> dummy (tx by height)
- *   T[account][hash] -> dummy (tx by account)
- *   P[account][hash] -> dummy (pending tx by account)
- *   M[account][time][hash] -> dummy (tx by time + account)
  *   H[account][height][hash] -> dummy (tx by height + account)
- *   C[account][hash][index] -> dummy (coin by account)
  *   b[height] -> block record
+ *
+ *   Unconfirmed
+ *   -----------
+ *   w[time][count][hash] -> dummy (tx by time)
+ *   W[account][time][count][hash] -> dummy (tx by time + account)
+ *   e[hash] -> time (unconfirmed time for tx)
+ *   p[hash] -> dummy (pending flag)
+ *   P[account][hash] -> dummy (pending tx by account)
  */
 
 exports.txdb = {
   prefix: bdb.key('t', ['uint32']),
+
+  // Balance
   R: bdb.key('R'),
   r: bdb.key('r', ['uint32']),
+
+  // Transactions
   t: bdb.key('t', ['hash256']),
+  T: bdb.key('T', ['uint32', 'hash256']),
+  z: bdb.key('z', ['uint32', 'uint32']),
+  Z: bdb.key('Z', ['uint32', 'uint32', 'uint32']),
+  y: bdb.key('y', ['hash256']),
+  x: bdb.key('u', ['hash256']),
+
+  // Coins
   c: bdb.key('c', ['hash256', 'uint32']),
+  C: bdb.key('C', ['uint32', 'hash256', 'uint32']),
   d: bdb.key('d', ['hash256', 'uint32']),
   s: bdb.key('s', ['hash256', 'uint32']),
-  p: bdb.key('p', ['hash256']),
-  m: bdb.key('m', ['uint32', 'hash256']),
+
+  // Confirmed
+  g: bdb.key('g', ['uint32', 'uint32', 'uint32', 'hash256']),
+  G: bdb.key('G', ['uint32', 'uint32', 'uint32', 'uint32', 'hash256']),
   h: bdb.key('h', ['uint32', 'hash256']),
-  T: bdb.key('T', ['uint32', 'hash256']),
-  P: bdb.key('P', ['uint32', 'hash256']),
-  M: bdb.key('M', ['uint32', 'uint32', 'hash256']),
   H: bdb.key('H', ['uint32', 'uint32', 'hash256']),
-  C: bdb.key('C', ['uint32', 'hash256', 'uint32']),
-  b: bdb.key('b', ['uint32'])
+  b: bdb.key('b', ['uint32']),
+
+  // Unconfirmed
+  w: bdb.key('w', ['uint32', 'uint32', 'hash256']),
+  W: bdb.key('W', ['uint32', 'uint32', 'uint32', 'hash256']),
+  e: bdb.key('e', ['hash256']),
+  p: bdb.key('p', ['hash256']),
+  P: bdb.key('P', ['uint32', 'hash256'])
 };