Epic: Metrics collection/reporting for smart contract service

[david-bakin-sl]

Collect metrics on the behavior of the smart contract service in production, and visualize them.

• Visualization will be via the usual prometheus/grafana methods for (consensus|mirror|block) node; possibly supplemented by other Hashgraph/Hedera capabilities as available)

[N.B.: Tasks (i.e. issues) are listed twice here: Once segregated by "category" in the epic, and at the bottom in an official Github "tasklist" so that the issues can have the tracking button (back to this epic).]

Exploration document at internal Hashgraph Notions page (At this stage of the open source initiative we haven't yet migrated existing documentation to open access; that'll come eventually for this document)

• R&D investigation: Metrics for smart contract service from consensus node #15419 -
• R&D investigation: Approaches to get smart contract service metrics from _other_ than consensus node #16087 -
• Design doc

Metrics that can only be collected on consensus node:

• Metric for tracking throttled transactions #16035 - #TXs rejected because throttle limits hit
• feat: add smart contract service metrics for txs that fail pureCheck #16076 - #TXs rejected because pure checks fail
• % time on main tx thread spent in smart contracts service
  • First step is to see if combining per-tx-kind counts/sec with average ms per-tx-kind (existing metrics) will work

(Following sections need to be fleshed out:)

Metrics that can be collected from record/block stream:

• Obviously will be blocked waiting for block node; and how to do these metrics with the block stream needs to be explored (e.g., as a "plug in" to block node - a feature that doesn't yet exist - or as a separate service consuming the block stream)
• Doesn't pay to do it on the record stream: Quite an expensive development proposition and it'll be going away soon

Metrics that can be collected from the mirror node:

• TBD

Metrics that can be collected by play back of production events into an instrumented consensus node (as local node):

• TBD

Metrics that we can easily get from the consensus node (until we get them elsewhere):

• feat: SCSC metrics from consensus node (that we can later get elsewhere) #16088 <!~- feat: SCSC metrics from consensus node (that we can later get elsewhere) -->

[Given milestone as v0.56 because I have to pick something and that's what's available. But full implementation of this Epic may take awhile.]

Tasks

Preview Give feedback

1. R&D investigation: Approaches to get smart contract service metrics from _other_ than consensus node #16087
   Hedera Smart Contract Service +1
2. feat: add smart contract service metrics for txs that fail pureCheck #16076
   Hedera Smart Contract Service +1
   
3. feat: SCSC metrics from consensus node (that we can later get elsewhere) #16088
   Hedera Smart Contract Service +1