feat(metrics): B1 live apply-phase Prometheus gauges

[arkadianet]

Summary

• Add ApplyPhaseMetrics atomics around both process_block call sites so operators can see apply in progress without waiting for the next snapshot tick.
• Expose four gauges on /metrics (and matching ApiStatus fields): apply_in_progress, last_apply_duration_ms, last_applied_height, last_apply_age_ms.
• Wire live values through SnapshotReadState::status() at boot.

Test plan

• cargo fmt --all -- --check
• cargo clippy -p ergo-sync -p ergo-api -p ergo-node --all-targets --all-features -- -D warnings
• cargo test -p ergo-sync --lib apply_phase
• cargo test -p ergo-node --lib api_bridge
• Sample ergo-api fixture tests (router_layout, serve_shutdown, …)
• After deploy: curl -s localhost:9053/metrics | grep ergo_node_apply

Made with Cursor

Summary by CodeRabbit

• New Features

  • Added node telemetry showing whether full-block application is in progress.
  • Added metrics for the duration, height, and age of the most recent block application.
  • Exposed these metrics through the Prometheus /metrics endpoint and node status data.

• Tests

  • Updated API and synchronization coverage to validate the expanded status information and apply-phase tracking.

[coderabbitai]

Warning

Review limit reached

@arkadianet, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 57 seconds

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 305f79db-191f-43a1-b81c-4810c6a203df

📥 Commits

Reviewing files that changed from the base of the PR and between 85f9270 and 604bb1f.

📒 Files selected for processing (47)

• ergo-api/tests/blockchain_balance_routes.rs
• ergo-api/tests/blockchain_block_routes.rs
• ergo-api/tests/blockchain_box_range_route.rs
• ergo-api/tests/blockchain_box_routes.rs
• ergo-api/tests/blockchain_byaddress_routes.rs
• ergo-api/tests/blockchain_byergotree_routes.rs
• ergo-api/tests/blockchain_indexed_height.rs
• ergo-api/tests/blockchain_scala_parity.rs
• ergo-api/tests/blockchain_template_routes.rs
• ergo-api/tests/blockchain_token_routes.rs
• ergo-api/tests/blockchain_transaction_range_route.rs
• ergo-api/tests/blockchain_tx_routes.rs
• ergo-api/tests/blockchain_unspent_byaddress_routes.rs
• ergo-api/tests/blocks_at_parity.rs
• ergo-api/tests/blocks_header_ids_parity.rs
• ergo-api/tests/blocks_id_parity.rs
• ergo-api/tests/blocks_modifier_parity.rs
• ergo-api/tests/blocks_proof_for_tx_parity.rs
• ergo-api/tests/compat_blocks_submit_route.rs
• ergo-api/tests/compat_submit_routes.rs
• ergo-api/tests/difficulty_history_route.rs
• ergo-api/tests/events_endpoint.rs
• ergo-api/tests/extra_index_router_walk.rs
• ergo-api/tests/fixtures/openapi_native.yaml
• ergo-api/tests/fixtures/openapi_v1.yaml
• ergo-api/tests/host_schema.rs
• ergo-api/tests/identity_schema.rs
• ergo-api/tests/indexer_status_endpoint.rs
• ergo-api/tests/mempool_overlay_oracle.rs
• ergo-api/tests/mempool_source_schema.rs
• ergo-api/tests/miner_stats_route.rs
• ergo-api/tests/nipopow_routes_parity.rs
• ergo-api/tests/openapi_native_runtime_mount.rs
• ergo-api/tests/recent_blocks_route.rs
• ergo-api/tests/router_layout.rs
• ergo-api/tests/serve_shutdown.rs
• ergo-api/tests/storage_rent_assets_route.rs
• ergo-api/tests/submit_routes.rs
• ergo-api/tests/v1_batch_routes.rs
• ergo-api/tests/v1_boxes_tokens_addresses_routes.rs
• ergo-api/tests/v1_chain_tx_routes.rs
• ergo-api/tests/v1_light_stats_diag_routes.rs
• ergo-api/tests/v1_mempool_routes.rs
• ergo-api/tests/v1_operator_routes.rs
• ergo-api/tests/v1_script_routes.rs
• ergo-node/src/api_bridge/tests.rs
• ergo-sync/src/apply_phase.rs

📝 Walkthrough

Walkthrough

Adds apply-phase metrics for block processing, propagates live values through node status, exposes four Prometheus gauges, and updates API test fixtures for the expanded ApiStatus structure.

Changes

Apply-phase telemetry

Layer / File(s)	Summary
Apply-phase metrics state ergo-sync/src/apply_phase.rs, ergo-sync/src/lib.rs	Defines atomic apply metrics, RAII completion handling, accessors, unit tests, and crate exports.
Executor apply instrumentation ergo-sync/src/executor/mod.rs	Records success and failure outcomes for block application attempts in executor paths.
Node status wiring ergo-node/src/api_bridge.rs, ergo-node/src/node/boot.rs, ergo-node/src/snapshot.rs, ergo-node/src/api_bridge/tests.rs, ergo-api/src/types.rs	Passes shared metrics into SnapshotReadState, overlays live values onto ApiStatus, and initializes status defaults.
API and Prometheus exposure ergo-api/src/server.rs, ergo-api/tests/*	Adds four apply-related Prometheus gauges and updates status fixtures across API tests.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SyncExecutor
  participant ApplyPhaseMetrics
  participant SnapshotReadState
  participant Prometheus
  SyncExecutor->>ApplyPhaseMetrics: record block apply outcome
  SnapshotReadState->>ApplyPhaseMetrics: read live metrics
  SnapshotReadState-->>Prometheus: provide ApiStatus values
  Prometheus->>Prometheus: render apply gauges

Loading

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 28.70% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately summarizes the main change: adding live apply-phase Prometheus gauges.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/metrics-apply-phase

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[arkadianet]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

🧹 Nitpick comments (3)

ergo-node/src/api_bridge.rs (1)

78-79: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Wiring looks correct; consider a dedicated regression test for the live overlay.

The overlay logic itself is correct (ordering, field mapping all line up with ApplyPhaseMetrics). However, no test in this cohort exercises status() with a non-default ApplyPhaseMetrics — api_bridge/tests.rs only ever passes ApplyPhaseMetrics::default(). A test that calls apply_phase.begin()/success(height) then asserts status().apply_in_progress/last_apply_duration_ms/last_applied_height/last_apply_age_ms reflect it would catch a future regression (e.g., a dropped field assignment) that the type system won't.

Happy to draft this test if useful.

Also applies to: 257-271, 398-408

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ergo-node/src/api_bridge.rs` around lines 78 - 79, Add a regression test in
the API bridge tests that uses a non-default ApplyPhaseMetrics, invokes begin()
and success(height), then calls status() and asserts apply_in_progress,
last_apply_duration_ms, last_applied_height, and last_apply_age_ms reflect the
live metrics overlay. Keep existing default-metrics coverage unchanged.

ergo-sync/src/apply_phase.rs (1)

21-36: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add #[must_use] to ApplyPhaseGuard to prevent silent misuse.

Nothing stops a future call site from writing self.apply_phase.begin(); as a bare statement — the guard would be constructed and immediately dropped, defeating the "live in-progress" gauge for that call without any compiler warning. Both current call sites correctly bind the guard, but this is cheap insurance against regressions in new call sites.

🛡️ Proposed fix

 /// RAII: sets `in_progress` for the duration of one `process_block` call.
+#[must_use]
 pub struct ApplyPhaseGuard<'a> {
     metrics: &'a ApplyPhaseMetrics,
     started: Instant,
     finished: bool,
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ergo-sync/src/apply_phase.rs` around lines 21 - 36, Add the #[must_use]
attribute to the ApplyPhaseGuard struct so callers are warned when begin()
returns a guard that is immediately discarded. Leave ApplyPhaseMetrics::begin
and the guard’s existing behavior unchanged.

ergo-api/tests/blocks_proof_for_tx_parity.rs (1)

40-66: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoff

Consider a shared ApiStatus test default to avoid repeating this literal ~40 times.

This PR touches every API test file to add four fields to the same struct literal. Deriving Default for ApiStatus (or adding a test_default() helper in ergo-api) would let stubs use ..ApiStatus::default() for unused fields, so future field additions don't require a fan-out edit across every test.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ergo-api/tests/blocks_proof_for_tx_parity.rs` around lines 40 - 66, Add a
shared default construction path for ApiStatus, preferably by deriving or
implementing Default in the ergo-api definition, then update the test stub’s
status method to initialize only its meaningful overrides and fill remaining
fields via ApiStatus::default(). Apply the shared pattern across affected API
test stubs so future ApiStatus fields do not require repeated literal updates.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@ergo-api/tests/blocks_proof_for_tx_parity.rs`:
- Around line 40-66: Add a shared default construction path for ApiStatus,
preferably by deriving or implementing Default in the ergo-api definition, then
update the test stub’s status method to initialize only its meaningful overrides
and fill remaining fields via ApiStatus::default(). Apply the shared pattern
across affected API test stubs so future ApiStatus fields do not require
repeated literal updates.

In `@ergo-node/src/api_bridge.rs`:
- Around line 78-79: Add a regression test in the API bridge tests that uses a
non-default ApplyPhaseMetrics, invokes begin() and success(height), then calls
status() and asserts apply_in_progress, last_apply_duration_ms,
last_applied_height, and last_apply_age_ms reflect the live metrics overlay.
Keep existing default-metrics coverage unchanged.

In `@ergo-sync/src/apply_phase.rs`:
- Around line 21-36: Add the #[must_use] attribute to the ApplyPhaseGuard struct
so callers are warned when begin() returns a guard that is immediately
discarded. Leave ApplyPhaseMetrics::begin and the guard’s existing behavior
unchanged.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: c6161ed7-deae-4f88-9a13-94a1b499fc3d

📥 Commits

Reviewing files that changed from the base of the PR and between 94b037c and 85f9270.

📒 Files selected for processing (52)

• ergo-api/src/server.rs
• ergo-api/src/types.rs
• ergo-api/tests/blockchain_balance_routes.rs
• ergo-api/tests/blockchain_block_routes.rs
• ergo-api/tests/blockchain_box_range_route.rs
• ergo-api/tests/blockchain_box_routes.rs
• ergo-api/tests/blockchain_byaddress_routes.rs
• ergo-api/tests/blockchain_byergotree_routes.rs
• ergo-api/tests/blockchain_indexed_height.rs
• ergo-api/tests/blockchain_scala_parity.rs
• ergo-api/tests/blockchain_template_routes.rs
• ergo-api/tests/blockchain_token_routes.rs
• ergo-api/tests/blockchain_transaction_range_route.rs
• ergo-api/tests/blockchain_tx_routes.rs
• ergo-api/tests/blockchain_unspent_byaddress_routes.rs
• ergo-api/tests/blocks_at_parity.rs
• ergo-api/tests/blocks_header_ids_parity.rs
• ergo-api/tests/blocks_id_parity.rs
• ergo-api/tests/blocks_modifier_parity.rs
• ergo-api/tests/blocks_proof_for_tx_parity.rs
• ergo-api/tests/compat_blocks_submit_route.rs
• ergo-api/tests/compat_submit_routes.rs
• ergo-api/tests/difficulty_history_route.rs
• ergo-api/tests/events_endpoint.rs
• ergo-api/tests/extra_index_router_walk.rs
• ergo-api/tests/host_schema.rs
• ergo-api/tests/identity_schema.rs
• ergo-api/tests/indexer_status_endpoint.rs
• ergo-api/tests/mempool_overlay_oracle.rs
• ergo-api/tests/mempool_source_schema.rs
• ergo-api/tests/miner_stats_route.rs
• ergo-api/tests/nipopow_routes_parity.rs
• ergo-api/tests/openapi_native_runtime_mount.rs
• ergo-api/tests/recent_blocks_route.rs
• ergo-api/tests/router_layout.rs
• ergo-api/tests/serve_shutdown.rs
• ergo-api/tests/storage_rent_assets_route.rs
• ergo-api/tests/submit_routes.rs
• ergo-api/tests/v1_batch_routes.rs
• ergo-api/tests/v1_boxes_tokens_addresses_routes.rs
• ergo-api/tests/v1_chain_tx_routes.rs
• ergo-api/tests/v1_light_stats_diag_routes.rs
• ergo-api/tests/v1_mempool_routes.rs
• ergo-api/tests/v1_operator_routes.rs
• ergo-api/tests/v1_script_routes.rs
• ergo-node/src/api_bridge.rs
• ergo-node/src/api_bridge/tests.rs
• ergo-node/src/node/boot.rs
• ergo-node/src/snapshot.rs
• ergo-sync/src/apply_phase.rs
• ergo-sync/src/executor/mod.rs
• ergo-sync/src/lib.rs