Is there a way to bulk/batch fetching of transactions across multiple blocks?

[thehenrytsai]

I am the lead developer in the Sidetree project, currently bcoin API is used to scan every transaction for Sidetree specific data. This means we make one API call per block which results in 100s of 1000s of network calls (and will be in the millions), which is rather inefficient, time consuming and not always reliable.

Is there a way to bulk/batch fetching of transactions across multiple or a range of blocks?

[pinheadmz]

What specific API call are you making each block? And what are the 100s/1000s of network calls that result?

If you can listen for block events from the FullNode, that event will deliver a block object containing all its transactions. A guide about bcoin events is here: https://bcoin.io/guides/events.html

Also you're kind of just in time with your adoption of bcoin, we have just merged a massive improvement to the indexer module: #758 @braydonf may be able to address your specific needs a bit more acutely about the indexer.

And finally, bcoin has a plugin architecture, you could run your full node with a plugin that emits a special event whenever an OP_RETURN transaction is added to the mempool, etc. I plan on going through the Sidetree implementation this week and looking for other ways to help :-)

[thehenrytsai]

What specific API call are you making each block? And what are the 100s/1000s of network calls that result?

RPC call:
{ "method": "getblockbyheight",
"params": [blockNumber, true, true]}

And finally, bcoin has a plugin architecture

This is interesting, can you expose API endpoints in the custom plugin with this plugin architecture? We used to do exactly this in bitcored, that is: have a plugin parsing every block for Sidetree data, and expose a custom API for external consumption), it works super efficiently but we moved away from it as bitcored appears to have removed the plugin model in their recent code revamp beyond Node v9.

We are currently using a pull model since a minute delay in our specific usage is acceptable and it keeps all the processing logic in one place (external to bcoin), but totally open to take advantage of the plugin architecture if that's the main suggested method for improving perf in this case.

[pinheadmz]

So when you say 1000s of network calls, you mean like during initial sidetree sync? Because the application will have to request each block one at a time from bcoin to check for all sidetree data?

Or are there 1000s of networks calls you have to make in response to each block?

[tynes]

can you expose API endpoints in the custom plugin with this plugin architecture?

Yes you can. A plugin has access to the node and its http endpoints, and you can define additional endpoints yourself. The wallet was designed as a plugin, see here for the implementation: https://github.com/bcoin-org/bcoin/blob/master/lib/wallet/plugin.js

A plugin needs an id property and an init method and you can specify plugins at runtime with --plugins

[thehenrytsai]

@pinheadmz,

So when you say 1000s of network calls, you mean like during initial sidetree sync?

Yes, would love to speed up initial startup/sync time by reducing the number of network calls.

Because the application will have to request each block one at a time from bcoin to check for all sidetree data?

Yes.

Or are there 1000s of networks calls you have to make in response to each block?

No.

[pinheadmz]

@thehenrytsai If you have a chance, could you post the txid of a bitcoin transaction with sidetree content? I'm going to try to write a simple OP_RETURN scanning indexer plugin and I want to see how your txs are structured -- looks like the prefix is sidetree: ? Which would appear in hex as 73696465747265653a ?

[pinheadmz]

@thehenrytsai I built a proof-of-concept OP_RETURN sidetree indexer as bcoin plugin:

https://github.com/pinheadmz/opreturn-indexer

When launched, it will maintain a separate index (even after Full Node is synced) of sidetree transactions that can then be fetched by range of block heights. Take a look and let me know if this is the kind of thing you're looking for. We can chat on slack as well if you have more specific questions or needs.

[braydonf]

Closed as looks like the question is answered by using a plugin. However if it's not please feel free to reopen.

As an additional note: The RPC method getblockbyheight will return a JSON result and a raw format. If the raw format is used for getblockbyheight, there could be an optimization that would directly read the block from disk (skipping deserialization/serialization). I have seen this optimization have about a 20x improvement to Bitcoin Core, it would likely be similar for bcoin.