Ensure predictable latency to apply chunks

[aborg-dev]

This is a tracking issue for a top-level goal to ensure that 99-percentile of apply chunk latency across all shards is below 800ms on the recommended validator hardware on mainnet.

Setup

Beta Give feedback

1. Setup a validator node on mainnet to measure apply chunk latency
2. Setup logging for outliers
3. Setup Grafana dashboard for tracking them

Workloads

Beta Give feedback

1. Tune sweat.claim workload to saturate chunk capacity #11145
   +0
2. Add ref.finance Locust workload #11169
   +0

[aborg-dev]

Here is the VM Instance: https://console.cloud.google.com/compute/instancesDetail/zones/europe-west4-a/instances/crt-mainnet?hl=en&project=nearone-crt. It is using this terraform script: https://github.com/Near-One/infra-ops/pull/64

It is currently downloading blocks

Apr 24 13:00:13 crt-mainnet neard[6014]: 2024-04-24T13:00:13.293735Z  INFO stats: #117520067 Downloading blocks 17.79% (2519 left; at 117520067) 32 peers ⬇ 10.0 MB/s ⬆ 4.23 MB/s 1.00 bps 958 Tgas/s CPU: 472%, Mem: 9.67 GB

I believe it should automatically appear on nearone Grafana, I'll share the link when it does.

[aborg-dev]

The node has been provisioned and is now running, here are the relevant Grafana dashboards:

• Chunk apply latency: https://nearone.grafana.net/goto/VZp4kpfSR?orgId=1
• Blockchain utilization: https://nearone.grafana.net/goto/MHO7kpfSR?orgId=1

I already see some long chunk apply latencies on shards 2, 3 and 5.

I think we can start with 2s threshold for alert, I see 2 such events on shard 5 in the last 6 hours. I'll add some logging to the node to find out the problematic block heights.

[aborg-dev]

We decided to decommission the node and instead focus on the FT benchmark, so closing this issue.