feat(l1): rpc instrumentation and panels #5311

rodrigo-o · 2025-11-12T23:20:34Z

Motivation

Add rpc and engine instrumentation and panels to our dashboards

Description

This PR adds a couple of changes to profiling.rs and our dashboard:

Add a simple way to time async funcrions without relying on #instrument (which would be more complex for every rpc)
Add namespace as a field to the instrumentation spans
Set a namespace for the old ones (block_execution)
Make the breakdown panels work with the new changes
Add new RPC and Engine panels

…rofiling

…nstrumentation

rodrigo-o · 2025-11-13T00:04:31Z

metrics/provisioning/grafana/dashboards/common_dashboards/ethrex_l1_perf.json

              "disableTextWrap": false,
              "editorMode": "code",
-              "expr": "(\n  sum by (function_name) (\n    increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", function_name!~\"Block execution|Execute Block|Trie update\"}[$__range])\n  )\n)\nor\nlabel_replace(\n  sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", function_name=\"Block execution\"}[$__range]) )\n  - sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", function_name=~\"Storage read|Account read|Account code read|Block hash read\"}[$__range]) ),\n  \"function_name\",\"Block Execution (w/o reads)\",\"__name__\",\".*\"\n)\nor\nlabel_replace(\n  sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", function_name=\"Execute Block\"}[$__range]) )\n  - sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", function_name=\"Block execution\"}[$__range]) ),\n  \"function_name\",\"Merkleization Diff\",\"__name__\",\".*\"\n)",
+              "expr": "(\n  sum by (function_name) (\n    increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", namespace!~\"rpc|engine\", function_name!~\"Block execution|Execute Block|Trie update\"}[$__range])\n  )\n)\nor\nlabel_replace(\n  sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", namespace!~\"rpc|engine\", function_name=\"Block execution\"}[$__range]) )\n  - sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", namespace!~\"rpc|engine\", function_name=~\"Storage read|Account read|Account code read|Block hash read\"}[$__range]) ),\n  \"function_name\",\"Block Execution (w/o reads)\",\"__name__\",\".*\"\n)\nor\nlabel_replace(\n  sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", namespace!~\"rpc|engine\", function_name=\"Execute Block\"}[$__range]) )\n  - sum( increase(function_duration_seconds_sum{job=\"$job\", instance=~\"$instance(:\\\\d+)?$\", namespace!~\"rpc|engine\", function_name=\"Block execution\"}[$__range]) ),\n  \"function_name\",\"Merkleization Diff\",\"__name__\",\".*\"\n)",


Instead of matching with the newly added block_execution namespace we are matching against the 2 additional ones (RPC and engine) this is to support the current running nodes when this changes are deployed to our centralized grafana instance. It avoids breaking the current Execution Breakdown for nodes that doesn'ta have this change and here is an issue to made this change in the following days: #5312

rodrigo-o · 2025-11-13T00:06:15Z

crates/networking/rpc/rpc.rs

+        record_async_duration(namespace, req.method.as_str(), async move {
+            request.handle(context).await
+        })


We might want to have this behind the metrics feature flag maybe? I'll take a look at how the rest of the instrumentation was working with the previous breakdown panels and in any case create an issue to follow-up on this.

github-actions · 2025-11-13T00:11:32Z

Lines of code report

Total lines added: 103
Total lines removed: 0
Total lines changed: 103

Detailed view

+-----------------------------------------------+-------+------+
| File                                          | Lines | Diff |
+-----------------------------------------------+-------+------+
| ethrex/crates/blockchain/blockchain.rs        | 1313  | +15  |
+-----------------------------------------------+-------+------+
| ethrex/crates/blockchain/metrics/profiling.rs | 110   | +55  |
+-----------------------------------------------+-------+------+
| ethrex/crates/blockchain/vm.rs                | 124   | +20  |
+-----------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/rpc.rs           | 843   | +8   |
+-----------------------------------------------+-------+------+
| ethrex/crates/vm/backends/mod.rs              | 171   | +5   |
+-----------------------------------------------+-------+------+

Copilot

Pull Request Overview

This PR adds comprehensive RPC and Engine API instrumentation to improve observability and performance monitoring. The changes introduce a namespace-based profiling system that categorizes metrics by their source (block_execution, rpc, engine), along with new Grafana dashboard panels to visualize these metrics.

Key Changes

Introduced record_async_duration helper function for timing async operations without requiring #[instrument] attributes
Added namespace field to all instrumentation spans for better metric categorization
Created new RPC and Engine API dashboard panels showing request rates, latency, and method breakdowns

Reviewed Changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
crates/blockchain/metrics/profiling.rs	Added namespace support via `NamespaceVisitor`, implemented `record_async_duration` for async timing, and updated histogram to include namespace label
crates/networking/rpc/rpc.rs	Integrated `record_async_duration` in RPC handler to instrument all RPC/Engine calls with appropriate namespace
crates/networking/rpc/Cargo.toml	Added ethrex-metrics dependency for profiling functionality
crates/blockchain/blockchain.rs	Added namespace="block_execution" to Execute Block, Trie update, and Block DB update spans
crates/blockchain/vm.rs	Added namespace="block_execution" to database read operations (Account, Storage, Block hash, Account code)
crates/vm/backends/mod.rs	Added namespace="block_execution" to Block execution span
Cargo.toml	Added ethrex-metrics to workspace dependencies
Cargo.lock	Updated dependency graph for ethrex-rpc with ethrex-metrics
metrics/provisioning/grafana/dashboards/common_dashboards/ethrex_l1_perf.json	Added RPC and Engine API panels with request rates, latency charts, and method breakdowns; updated existing queries to filter by namespace

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

metrics/provisioning/grafana/dashboards/common_dashboards/ethrex_l1_perf.json

crates/blockchain/metrics/profiling.rs

metrics/provisioning/grafana/dashboards/common_dashboards/ethrex_l1_perf.json

crates/blockchain/metrics/profiling.rs

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

crates/blockchain/metrics/profiling.rs

MegaRedHand

LGTM

…mespace

**Motivation** Add rpc and engine instrumentation and panels to our dashboards  **Description** This PR adds a couple of changes to profiling.rs and our dashboard: - Add a simple way to time async funcrions without relying on `#instrument` (which would be more complex for every rpc) - Add namespace as a field to the instrumentation spans - Set a namespace for the old ones (`block_execution`) - Make the breakdown panels work with the new changes - Add new RPC and Engine panels  <img width="2551" height="780" alt="image" src="https://github.com/user-attachments/assets/3b9feead-fc21-4115-b91b-118157477f18" />  --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

**Motivation** Document the new execution breakdown and RPC panels in the dashboard as well as other metric related docs **Description** This PR documents both #5220 and #5311 additions, so we might want to merge #5311 first. It updates the gap analysis, the actual dashboard doc and some metrics related docs. It also sets prometheus scrape to 5s as already done today on our centralized grafana instance. Quick [link](https://github.com/lambdaclass/ethrex/blob/update-dashboard-documentation/docs/developers/l1/dashboards.md) to check the updated dashboard doc out with the viewer --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

rodrigo-o added 20 commits November 7, 2025 19:01

add namespace to profiling

8e6dc0b

initial testing of instumentation

5b5e3d6

remove rpc namespaced instrumentation from breakdown panels

042b266

initial rpc dashboards

e873b85

made some organization changes in the dashboard

cc97128

make rpc dashboard repeatable by instance

14dbeaf

Merge branch 'main' into rpc-instrumentation-and-panels

01bab1e

separate Engine API from RPC API in instrumentation

6dc47b9

change total sum by average in pie charts of rpc

d0885bb

move from increase to rate for calculating the avg

6f82153

go without any increase or rate

1b6cefa

rework panels for engine api

c7d20d6

reworked the RPC api panels

fe92933

Added instances to the rpc and engine panels

bc00add

remove unintended overrides

86fccd8

remove intemediary doc

5968fd3

initial rework of the instrumentation to avoid complex observers in p…

f75a0ac

…rofiling

go back to use just call to avoid most of rpc.rs changes made

6c4a9f5

Add namespace to profiling and explicitly set it up in the previous i…

024b3f7

…nstrumentation

fixed an issue with the way namespace is send through #instrument

c51eeea

rodrigo-o mentioned this pull request Nov 13, 2025

Make the break execution panels match the newly added instrumentation namespace #5312

Open

rodrigo-o commented Nov 13, 2025

View reviewed changes

Merge branch 'main' into rpc-instrumentation-and-panels

35d9cbd

enhanced record_debug and remove magic namespacing given targets

28ec41f

github-actions bot added the L1 Ethereum client label Nov 13, 2025

github-project-automation bot added this to ethrex_l1 Nov 13, 2025

format

6e5b041

rodrigo-o marked this pull request as ready for review November 13, 2025 13:41

rodrigo-o requested a review from a team as a code owner November 13, 2025 13:41

Copilot AI review requested due to automatic review settings November 13, 2025 13:41

ethrex-project-sync bot moved this to In Review in ethrex_l1 Nov 13, 2025

Copilot started reviewing on behalf of rodrigo-o November 13, 2025 13:42 View session

Copilot finished reviewing on behalf of rodrigo-o November 13, 2025 13:43

Copilot AI reviewed Nov 13, 2025

View reviewed changes

MegaRedHand reviewed Nov 13, 2025

View reviewed changes

crates/blockchain/metrics/profiling.rs Outdated Show resolved Hide resolved

crates/blockchain/metrics/profiling.rs Show resolved Hide resolved

rodrigo-o and others added 2 commits November 13, 2025 14:54

apply suggestions

f6a401b

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

missed suggestions

cd1534d

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

MegaRedHand reviewed Nov 13, 2025

View reviewed changes

crates/blockchain/metrics/profiling.rs Outdated Show resolved Hide resolved

updated cargo lock for l2 quote-gen

2948d97

MegaRedHand approved these changes Nov 13, 2025

View reviewed changes

MegaRedHand mentioned this pull request Nov 13, 2025

Simplify FunctionProfilingLayer #5327

Open

rodrigo-o added 5 commits November 13, 2025 15:38

remove Cow given that it was already cloning

39bd789

Add comment about why we use String instead of &'static str in the na…

5b167ab

…mespace

limit the life of exetension to the let timer block

e890efa

modify 5m ranges to the actual range selected in some panels

bd95d31

Merge branch 'main' into rpc-instrumentation-and-panels

89f1a09

rodrigo-o mentioned this pull request Nov 14, 2025

docs(l1): update dashboard documentation #5334

Merged

jrchatruc approved these changes Nov 14, 2025

View reviewed changes

jrchatruc added this pull request to the merge queue Nov 14, 2025

Merged via the queue into main with commit 9fc41e7 Nov 14, 2025
49 of 55 checks passed

jrchatruc deleted the rpc-instrumentation-and-panels branch November 14, 2025 20:48

github-project-automation bot moved this from In Review to Done in ethrex_l1 Nov 14, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(l1): rpc instrumentation and panels #5311

feat(l1): rpc instrumentation and panels #5311

Uh oh!

rodrigo-o commented Nov 12, 2025 •

edited

Loading

Uh oh!

rodrigo-o Nov 13, 2025

Uh oh!

rodrigo-o Nov 13, 2025

Uh oh!

github-actions bot commented Nov 13, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MegaRedHand left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

feat(l1): rpc instrumentation and panels #5311

feat(l1): rpc instrumentation and panels #5311

Uh oh!

Conversation

rodrigo-o commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rodrigo-o Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

rodrigo-o Nov 13, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Lines of code report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

MegaRedHand left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rodrigo-o commented Nov 12, 2025 •

edited

Loading

github-actions bot commented Nov 13, 2025 •

edited

Loading