Skip to content

Comments

Expose worker metrics via OTEL#10

Merged
masih merged 3 commits intomainfrom
masih/otel-metrics
Aug 13, 2025
Merged

Expose worker metrics via OTEL#10
masih merged 3 commits intomainfrom
masih/otel-metrics

Conversation

@masih
Copy link
Contributor

@masih masih commented Aug 6, 2025

Expose the worker load metrics via OTEL and add prometheus exporter to CLI. The metrics exposed provides:

  • Send latency histogram
  • Receipt latency histogram
  • Worker queue length

Note that the latency histograms can be used to measure the rate of requests per second among other throughput analysis. The metrics are tagged by chain ID, endpoint and worker ID to facilitate drilldown in results.

attribute.Int64("chain_id", w.chainID),
statusAttrFromError(_err)),
)
}(time.Now())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might have an issue where we will see large variance based on whether slow=true or false When slow=true, we wait for the tx to be committed in a block, whereas slow=false will allow for tx to enter mempool and that's it. We currently use slow=true for easier nonce management, but curious @stevenlanders 's thoughts on how we might want to consider this factor in this metric. Since in one case it will measure round trip time for tx send until fully executed and when slow=false its just time til mempool inclusion.

Copy link
Contributor Author

@masih masih Aug 8, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, we could simply tag the metrics by slow to differentiate?

masih added 2 commits August 11, 2025 20:11
Expose the worker load metrics via OTEL and add prometheus exporter to
CLI. The metrics exposed provides:
* Send latency histogram
* Receipt latency histogram
* Worker queue length

Note that the latency histograms can be used to measure the rate of
requests per second among other throughput analysis. The metrics are
tagged by chain ID, endpoint and worker ID to facilitate drilldown in
results.
@masih masih force-pushed the masih/otel-metrics branch from f1efcaf to df1b795 Compare August 11, 2025 19:12
Copy link
Collaborator

@stevenlanders stevenlanders left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm - i think it's okay not to worry about slow vs. not within the sender. This thing should be pretty dumb and just report what it sees.

@masih masih merged commit 841e116 into main Aug 13, 2025
2 checks passed
@masih masih deleted the masih/otel-metrics branch August 13, 2025 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants