Indexer fixes and improvements

[braydonf]

This adds a new module indexer for indexing blocks, and includes several fixes and improvements:

• Reduces the size of the txindex from being around 100% of all blocks, down to around 10% of the blocks (Disk I/O efficiency improvements #585).
• Fixes DoS by CPU and memory exhaustion for the addrindex and reduces disk usage (CPU exhaustion with address index HTTP API queries #589).
• Will correctly distinguish between p2pkh and p2wpkh addresses for the address index, instead of combining the results that use the same public key (Output descriptor not included for address index #738).
• Adds an ability to enable the indexes after the initial block download (Interface for indexing plugins #422).
• Indexes have been moved into a separate database, and can be disabled or regenerated asynchronously without changing the chain database (Interface for indexing plugins #422 and Memory exhaustion crash with full node startup #682).
• Includes an indexing interface for extensibility (Interface for indexing plugins #422).
• Resolves issues with duplicate transactions and coins in the mempool (duplicate copies of transactions and UTXOs created from reorganization #499 and Duplicate transactions received when requesting them from bcoin node #594).
• Adds clarification on the status of the indexers (missing transactions for address index #734).

The following issues are closed:

• Closes: Output descriptor not included for address index #738
• Closes: missing transactions for address index #734
• Closes: Duplicate transactions received when requesting them from bcoin node #594
• Closes: CPU exhaustion with address index HTTP API queries #589
• Closes: Disk I/O efficiency improvements #585
• Closes: duplicate copies of transactions and UTXOs created from reorganization #499
• Closes: Interface for indexing plugins #422

Includes commits from @tuxcanfly, @braydonf and @nodar-chkuaselidze.

[tynes]

Could you add migrate/chaindb5to6.js to migrate/latest?

[braydonf]

I'm actually thinking we can combine migrate/chain4to5.js and migrate/chain5to6.js, that will do both. In that we can likely update migrate/latest, from what I remember it's not current.

[tynes]

It would be really nice to have some indexer information returned from the API. A call to GET / on the node could return an additional field:

"indexer": {
  "addr": {
      "enabled": true,
      "height": 40000
    },
   "tx" : {
      "enabled": false,
      "height" null
    }
}

As @braydonf pointed out, endpoints will return [] if the indexers are not enabled

If an Indexer has some extra methods that enable this data collection (something like indexer.isEnabled, indexer.bestHeight) and all types of Indexers implement those methods, then this object could be dynamically created and updated if additional indexers are added, ie by plugins.

Would be happy to open an issue if this is out of context for this particular PR

[braydonf]

@tynes The indexer status has been added, along with the logging of the current status of the indexer, this should help with checking if the indexing is enabled.

[braydonf]

Rebased on latest master branch. Further documentation updates are at bcoin-org/bcoin-org.github.io#145.

[tuxcanfly]

OK.