Reflect APM collected metrics in Observability Overview

[graphaelli]

Describe the feature: When collecting metrics only with APM the overview page reports no data:

while the metrics app is able to show a useful inventory:

Describe a specific use case for the feature:

Ideally the inventory on the overview page can be powered by data already collected by APM. If that's not possible, what data is expected to make this inventory more useful?

cc @alex-fedotyev @sorantis @Mpdreamz

[elasticmachine]

Pinging @elastic/logs-metrics-ui (Team:logs-metrics-ui)

[sorantis]

Gil, I think the Metrics app can show APM data only with the apm-* is set in the app's Settings page. Not sure if what's rendered on the O11y overview page takes that into account. cc @simianhacker @jasonrhodes

[simianhacker]

My guess is that it's missing the system module fields so they don't show up properly because it's not the dataset we are expecting.

[alex-fedotyev]

@graphaelli - I tried this and I think this is a bigger ask than just Kibana.
Kibana isn't aware of the situations when apm-* and metrics-* both have collected data about the same entities and renders them as duplicates.
We would need to figure out how to address problems of data duplication and visualization together.

[graphaelli]

Sorry, notifications on your responses were filtered out. I should have been more specific that this is data collected with the APM Integration, so the data goes into data streams matching the metrics ui's default patterns.

[graphaelli]

@simianhacker can you share more details on what those fields are? An example query made by the overview page would be great.

[simianhacker]

Here is the example query:

{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
      {
        "range": {
          "@timestamp": {
            "gte": 1629225343115,
            "lte": 1629225643115,
            "format": "epoch_millis"
          }
        }
      },
      {
        "exists": {
          "field": "host.name"
        }
      }
      ]
    }
  },
  "aggs": {
    "groupings": {
      "composite": {
        "size": 2000,
        "sources": [
        {
          "groupBy0": {
            "terms": {
              "field": "host.name"
            }
          }
        }
        ]
      },
      "aggs": {
        "histogram": {
          "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "60s",
            "offset": "-17115ms",
            "extended_bounds": {
              "min": 1629225343115,
              "max": 1629225643115
            }
          },
          "aggregations": {
            "cpu_user": {
              "avg": {
                "field": "system.cpu.user.pct"
              }
            },
            "cpu_system": {
              "avg": {
                "field": "system.cpu.system.pct"
              }
            },
            "cpu_cores": {
              "max": {
                "field": "system.cpu.cores"
              }
            },
            "cpu": {
              "bucket_script": {
                "buckets_path": {
                  "user": "cpu_user",
                  "system": "cpu_system",
                  "cores": "cpu_cores"
                },
                "script": {
                  "source": "(params.user + params.system) / params.cores",
                  "lang": "painless"
                },
                "gap_policy": "skip"
              }
            },
            "__metadata__": {
              "top_metrics": {
                "size": 1,
                "metrics": [
                {
                  "field": "host.name"
                },
                {
                  "field": "host.ip"
                }
                ],
                "sort": {
                  "@timestamp": "desc"
                }
              }
            }
          }
        }
      }
    }
  }
}

[graphaelli]

Thanks! I guess that table doesn't like when the cpu information isn't provided then? This query returns something like this against metrics-apm*:

     "buckets" : [
        {
          "key" : {
            "groupBy0" : "2ebc7b9f76e2"
          },
          "doc_count" : 2877,
          "histogram" : {
            "buckets" : [
              {
                "key_as_string" : "2021-08-17T19:19:00.000Z",
                "key" : 1629227940000,
                "doc_count" : 19,
                "cpu_user" : {
                  "value" : null
                },
                "__metadata__" : {
                  "top" : [
                    {
                      "sort" : [
                        "2021-08-17T19:19:44.629Z"
                      ],
                      "metrics" : {
                        "host.name" : "2ebc7b9f76e2",
                        "host.ip" : "172.19.0.8"
                      }
                    }
                  ]
                },
                "cpu_cores" : {
                  "value" : null
                },
                "cpu_system" : {
                  "value" : null
                }
              },

[graphaelli]

FWIW I did try to update_by_query values for system.cpu.user.pct, system.cpu.system.pct, system.cpu.cores but the charts still didn't show anything, time to pass this off to apm ui team to work with you on this @simianhacker

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

Did we decide whether APM data should be filtered out, or if we actually want to make use of it? It sounds like if it's not filtered out it could somehow count double. OTOH if there's no metrics data it might be nice to fall back to APM data to populate visualisations.

@simianhacker We can sync on this if you want. I can help out with getting you some APM data for you to test with if needed.