Skip to content

Conversation

@rodrigo-o
Copy link
Collaborator

@rodrigo-o rodrigo-o commented Nov 6, 2025

Motivation

We needed more granular panels to analyze the breakdown of execution for performance decisions.

Description

This PR use the current instrumentation to:

  • Update the pie char of execution
  • Add a diff measurement between execution and merkleization to understand the work of the execute_block_pipeline function
  • Add a deagregated series by block showing the different instrumentations available today
  • Maintain all of them in the previous Block Execution Breakdown row and repeat them vertically across instances
image

Note for running it locally:

After #5088 we needed to re-labeling our local instances to avoid duplicated panels that repeat across instances. This requires prometheus to be restarted to pick up the new labeling, one way to check this out is make look at intances, if ethereum-metrics-exporte appears there, we need to restart prometheus, only localhost should be there locally.

Bad (needs restarting prometheus) Good
image image

Next steps:

  • Naming is not optimal, but changing it would require breaking the panel for instances not updated to this PR. We want first to test it with all instances and make sure the panels are useful and change the names afterwards, here is an issue for tracking that: Rename function instrumentation to be consistent #5219

Closes #5218

@rodrigo-o rodrigo-o changed the title feat(l1): upgrade execution breakdown pie char and add new panels feat(l1): upgrade execution breakdown pie chart and add new panels Nov 6, 2025
}
}

#[instrument(level = "trace", name = "Block execution", skip_all)]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are not instrumenting other non-pipeline related functions, if we want to do so we might need to do it in a following PR.


/// Applies account updates based on the block's latest storage state
/// and returns the new state root after the updates have been applied.
#[instrument(level = "trace", name = "Trie update", skip_all)]
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was already instrumented at handle_merkleization, if it was intended to be part of the non-pipeline instrumentation, we need to do it in a follow-up.

@github-actions
Copy link

github-actions bot commented Nov 6, 2025

Lines of code report

Total lines added: 1
Total lines removed: 3
Total lines changed: 4

Detailed view
+-------------------------------------------+-------+------+
| File                                      | Lines | Diff |
+-------------------------------------------+-------+------+
| ethrex/crates/blockchain/blockchain.rs    | 1314  | +1   |
+-------------------------------------------+-------+------+
| ethrex/crates/storage/store.rs            | 1520  | -1   |
+-------------------------------------------+-------+------+
| ethrex/crates/storage/store_db/rocksdb.rs | 1595  | -1   |
+-------------------------------------------+-------+------+
| ethrex/crates/vm/backends/mod.rs          | 166   | -1   |
+-------------------------------------------+-------+------+

@github-actions github-actions bot added the L1 Ethereum client label Nov 6, 2025
@rodrigo-o rodrigo-o marked this pull request as ready for review November 6, 2025 21:36
@rodrigo-o rodrigo-o requested a review from a team as a code owner November 6, 2025 21:36
Copilot AI review requested due to automatic review settings November 6, 2025 21:36
@ethrex-project-sync ethrex-project-sync bot moved this to In Review in ethrex_l1 Nov 6, 2025
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR reorganizes tracing instrumentation for performance monitoring by moving the "Block DB update" trace span from the storage layer to the blockchain layer, and removes redundant instrumentation from the VM execution path. Additionally, it includes significant Grafana dashboard enhancements for better block execution breakdown visualization.

  • Relocated "Block DB update" instrumentation from rocksdb.rs to blockchain.rs::store_block for more accurate monitoring
  • Removed instrumentation from execute_block and apply_account_updates_batch to reduce overhead
  • Enhanced Grafana dashboard with collapsed block execution breakdown section containing multiple detailed panels

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
crates/vm/backends/mod.rs Removed #[instrument] from execute_block method
crates/storage/store_db/rocksdb.rs Removed inline tracing span for "Block DB update"
crates/storage/store.rs Removed #[instrument] from apply_account_updates_batch and cleaned up unused instrument import
crates/blockchain/blockchain.rs Added #[instrument] to store_block method to track "Block DB update"
metrics/provisioning/grafana/dashboards/common_dashboards/ethrex_l1_perf.json Enhanced dashboard with collapsed block execution breakdown section, improved queries for execution vs merkleization analysis, and added deaggregated block-by-block execution visualization

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@rodrigo-o rodrigo-o added this pull request to the merge queue Nov 7, 2025
Merged via the queue into main with commit 5bbfc65 Nov 7, 2025
39 checks passed
@rodrigo-o rodrigo-o deleted the execution-breakdown-by-block branch November 7, 2025 20:19
@github-project-automation github-project-automation bot moved this from In Review to Done in ethrex_l1 Nov 7, 2025
xqft pushed a commit that referenced this pull request Nov 11, 2025
…5220)

**Motivation**

We needed more granular panels to analyze the breakdown of execution for
performance decisions.

**Description**

This PR use the current instrumentation to:
- Update the pie char of execution
- Add a diff measurement between execution and merkleization to
understand the work of the `execute_block_pipeline` function
- Add a deagregated series by block showing the different
instrumentations available today
- Maintain all of them in the previous Block Execution Breakdown row and
repeat them vertically across instances

<img width="1910" height="705" alt="image"
src="https://github.com/user-attachments/assets/0af6dfda-41b3-46f0-8463-1494c6ea0c8b"
/>


**Note for running it locally:**

_After #5088 we needed to re-labeling our local instances to avoid
duplicated panels that repeat across instances. This requires prometheus
to be restarted to pick up the new labeling, one way to check this out
is make look at intances, if `ethereum-metrics-exporte` appears there,
we need to restart prometheus, only `localhost` should be there
locally._

| Bad (needs restarting prometheus)| Good |
| --- | --- |
| <img width="359" height="156" alt="image"
src="https://github.com/user-attachments/assets/7c4bfd32-f565-4a79-bea9-86094128bfcf"
/> | <img width="453" height="176" alt="image"
src="https://github.com/user-attachments/assets/8e8423ec-3830-461f-94ab-697bd5967968"
/> |


**Next steps:**

- Naming is not optimal, but changing it would require breaking the
panel for instances not updated to this PR. We want first to test it
with all instances and make sure the panels are useful and change the
names afterwards, here is an issue for tracking that: #5219

Closes #5218

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
github-merge-queue bot pushed a commit that referenced this pull request Nov 17, 2025
**Motivation**

Document the new execution breakdown and RPC panels in the dashboard as
well as other metric related docs

**Description**

This PR documents both #5220 and #5311 additions, so we might want to
merge #5311 first.

It updates the gap analysis, the actual dashboard doc and some metrics
related docs. It also sets prometheus scrape to 5s as already done today
on our centralized grafana instance.

Quick
[link](https://github.com/lambdaclass/ethrex/blob/update-dashboard-documentation/docs/developers/l1/dashboards.md)
to check the updated dashboard doc out with the viewer

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

Enhance grafana execution breakdown panels

4 participants