Fine performance metrics: Prometheus and Grafana

[crusaderky]

• Part of Fine performance metrics meta-issue #7665
• Blocked by Fine performance metrics: Store data on the Scheduler #7666

Export the metrics collected on the scheduler by #7666 to Prometheus.
For the sake of keeping everything additive, execute, gather-dep, get-data, and async spilling metrics should be kept separate.

Coiled-specific deliverable: implement new Grafana plots that use the above metrics.

[fjetter]

The current schema of the data we are storing would be helpful for this issue ahead of time. Essentially what we are storing in #7666

We are already documenting something like this for existing metrics, see https://distributed.dask.org/en/latest/prometheus.html

What is missing on the documentation page is information about

• What labels are exported by metric
• What data type (count, duration, ...)

[fjetter]

Right now, it's not entirely certain if we want to expose this as a timeseries so there is also room to simply attach this information to a Computation object

[fjetter]

@hendrikmakait @crusaderky I believe there was a meeting discussing this topic. Can either of you summarize the outcome of it here?

Ok, I just realize there is a summary here #7665 (comment)

Does this mean we will not put those metrics in prometheus for now? If so, I suggest to close this issue

[hendrikmakait]

I'd be fine with closing this ticket as not planned for now and reopening it at a later point in time. Providing E2E aggregates should be the priority for now.

[crusaderky]

Agreed