cleaning up and improving function metrics

[jerrypeng]

Motivation

Cleaning up and improving metrics collection and presentation for functions

Modifications

1. Add endpoints to get stats of function or function instances
2. add function cli for stats
3. redesigned MetricsData protobuf
4. add overall aggregates stats for functions

for example:

./bin/pulsar-admin functions stats --name py-test
{
  "receivedTotal": 1000,
  "processedSuccessfullyTotal": 1000,
  "systemExceptionsTotal": 0,
  "userExceptionsTotal": 0,
  "avgProcessLatency": 0.06142544746398926,
  "lastInvocation": 1542318672866,
  "instances": [
    {
      "instanceId": 1,
      "metrics": {
        "receivedTotal": 500,
        "processedSuccessfullyTotal": 500,
        "systemExceptionsTotal": 0,
        "userExceptionsTotal": 0,
        "avgProcessLatency": 0.06087827682495117,
        "lastInvocation": 1542318672866,
        "userMetrics": {
          "user_metric_metrics_1_sum": 1500.0,
          "user_metric_metrics_1_count": 500.0,
          "user_metric_metrics_2_sum": 7500.0,
          "user_metric_metrics_2_count": 1000.0
        }
      }
    },
    {
      "instanceId": 0,
      "metrics": {
        "receivedTotal": 500,
        "processedSuccessfullyTotal": 500,
        "systemExceptionsTotal": 0,
        "userExceptionsTotal": 0,
        "avgProcessLatency": 0.061972618103027344,
        "lastInvocation": 1542318672766,
        "userMetrics": {
          "user_metric_metrics_1_sum": 1500.0,
          "user_metric_metrics_1_count": 500.0,
          "user_metric_metrics_2_sum": 7500.0,
          "user_metric_metrics_2_count": 1000.0
        }
      }
    }
  ]
}

and for instances:

./bin/pulsar-admin functions stats --name test --instance-id 0
{
  "receivedTotal": 500,
  "processedSuccessfullyTotal": 500,
  "systemExceptionsTotal": 0,
  "userExceptionsTotal": 0,
  "avgProcessLatency": 0.06325435638427734,
  "lastInvocation": 1542320665090,
  "userMetrics": {
    "user_metric_metrics_1_sum": 1500.0,
    "user_metric_metrics_1_count": 500.0,
    "user_metric_metrics_2_sum": 7500.0,
    "user_metric_metrics_2_count": 1000.0
  }
}

Getting all function stats on worker:

curl localhost:8080/admin/v2/worker-stats/functionsmetrics
[
  {
    "name": "public/default/test:0",
    "metrics": {
      "receivedTotal": 0,
      "processedSuccessfullyTotal": 0,
      "systemExceptionsTotal": 0,
      "userExceptionsTotal": 0,
      "avgProcessLatency": 0.0,
      "lastInvocation": 0,
      "userMetrics": {}
    }
  },
  {
    "name": "public/default/py-test:1",
    "metrics": {
      "receivedTotal": 500,
      "processedSuccessfullyTotal": 500,
      "systemExceptionsTotal": 0,
      "userExceptionsTotal": 0,
      "avgProcessLatency": 0.059513092041015625,
      "lastInvocation": 1542320665189,
      "userMetrics": {
        "user_metric_metrics_1_sum": 1500.0,
        "user_metric_metrics_1_count": 500.0,
        "user_metric_metrics_2_sum": 7500.0,
        "user_metric_metrics_2_count": 1000.0
      }
    }
  }
]

Next step in improving metrics, will be to add some system stats that are time windowed

FYI this PR overlaps with #2982

[merlimat]

👍

[jerrypeng]

rerun integration tests

[srkukarni]

For all new functionality, could we collapse getInstanceStats and getFunctionStats into one implementation? I see a lot of duplicate code

[jerrypeng]

Can't really combine those two function since getFunctionInstanceStats returns the stats for instance with getFunctionStats returns stats overall and for all instances. Different objects are returned. What can be refactored / combined is the logic between the get stats functions and get status functions. I didn't want to refactor the logic there since there are already many changes in this PR as it is.

[srkukarni]

sounds good