[RFC]: Metrics Refactoring

[lxning]

Current TorchServe has two mechanisms to emit metrics.

1. Emit metrics to logs files in a StatsD like format by default .

In this case, both frontend and backend metrics are recorded in log file. However, the logs format is not standard StatsD format. They miss the metric type information (ie. counter, gauge, timer and so on). Users have to write regex to parse the log to build dashboard.

2. Emit Prometheus formatted metrics.

In this case, existing TorchServe only emits 3 metrics.

• ts_inference_requests_total
• ts_inference_latency_microseconds
• ts_queue_latency_microseconds

Users are not able to get model metrics and system metrics via metrics endpoint.

No central place to store Metrics definition

Existing TorchServe metrics definitions spread everywhere. It is difficult for users to know the available metrics.

Re-Design

TS_Metrics_Design.pdf

Sub tasks on frontend side

• TorchServe default metrics config yaml file #2133
• Frontend Metric Interface and Class  #2139
• Metrics Caching #2140
• migrate existing frontend metrics to the new cache solution #2141

### Tasks
- [ ] https://github.com/pytorch/serve/issues/2747
- [ ] https://github.com/pytorch/serve/issues/2794
- [ ] https://github.com/pytorch/serve/issues/2772
- [ ] https://github.com/pytorch/serve/issues/2795

[msaroufim]

For the requirements here need to make sure to include

• Adding full coverage for frontend and backend metrics with prometheus export
• Instructions to make it easy for anyone to export their own metrics to prometheus

[thimabru1010]

Yes, I only can see these 3 metrics in prometheus. I tried to change model log behavior with a log4j2.xml file, but with no success. How do I get system metrics in prometheus?

[jonhilgart22]

Custom metrics would be great. I'd like to add some histogram metrics for Prometheus to see latency percentiles.

[sharvil]

Refactoring the existing metrics implementation is a welcome change. After reading through the proposed design, I recommend against introducing the metrics.yaml file.

In our case, we have model-specific performance metrics that we'd like to log and monitor. The ideal scenario is for each model archive to manage its own metrics. As those archives are loaded / unloaded they would register / unregister their metrics and those changes would be reflected in the metrics exposed by the frontend.

Defining a global metrics schema in metrics.yaml would require us to update that file separately every time a model is loaded or unloaded. In addition, doing so is repetitive since the proposed MetricsCaching.addMetric method already needs us to specify all the properties that would be in metrics.yaml.