Elastic integration packages load assets like mappings to ensure the data is indexed in the best way and dashboards / visualisations and other assets to consume the data. Currently there is no testing in place that validates if all the fields indexed are needed and are indexed in the best way. The following is a potential proposal on how some additional validation can be added.

Side note: More details around the topic can be found in the presentation from @3kt (Slides, Recording).

Compare what is mapped and ingested vs used

Each dataset maps the field that are used in the fields.yml files. With the introduction of the dynamic ecs@mappings template in Elasticsearch, now only the fields which are not ECS have to be mapped. Lets quickly introduce the relevant APIs based on the metrics-system.cpu-default data stream.

• Mappings: To find out, what fields are mapped in an index, GET metrics-system.cpu-default/_mapping API can be used.
• Disk usage per field: After ingestion, to find out which field uses how much storage, used the analyze index disk usage API: POST metrics-system.cpu-default/_disk_usage?run_expensive_tasks=true
• Query usage: As soon as some query / visualisations are run, to get details on which fields are queried how, use the field usage stats API: GET metrics-system.cpu-default/_field_usage_stats
• Disk usage total: For completeness, there is also the stats API that gives overall disk usage: GET metrics-system.cpu-default/_stats

With the first 3 APIs, we can have for a single field what is mapped, what is stored and what is queried.

Example

As our example, we use the metrics-system.memory-default data stream and the field system.memory.actual.used.pct. This is used in the System Metrics Overview dashboard:

Fist we want to see how the field is mapped an we run GET metrics-system.memory-default/_mapping. The result is as following:

...
"system": {
  "properties": {
    "memory": {
      "properties": {
        "actual": {
          "properties": {
            "free": {
              "type": "long",
              "meta": {
                "unit": "byte"
              },
              "time_series_metric": "gauge"
            },
            "used": {
              "properties": {
                "bytes": {
                  "type": "long",
                  "meta": {
                    "unit": "byte"
                  },
                  "time_series_metric": "gauge"
                },
                "pct": {
                  "type": "scaled_float",
                  "meta": {
                    "unit": "percent"
                  },
                  "scaling_factor": 1000,
                  "time_series_metric": "gauge"
                }
              }
            }
          }
        },
...

The field is mapped as scaled_float and "time_series_metric": "gauge". Now lets have a look at the disk usage.

POST metrics-system.memory-default/_disk_usage?run_expensive_tasks=true

The following info is returned:

"system.memory.actual.used.pct": {
  "total": "818b",
  "total_in_bytes": 818,
  "inverted_index": {
    "total": "0b",
    "total_in_bytes": 0
  },
  "stored_fields": "0b",
  "stored_fields_in_bytes": 0,
  "doc_values": "818b",
  "doc_values_in_bytes": 818,
  "points": "0b",
  "points_in_bytes": 0,
  "norms": "0b",
  "norms_in_bytes": 0,
  "term_vectors": "0b",
  "term_vectors_in_bytes": 0,
  "knn_vectors": "0b",
  "knn_vectors_in_bytes": 0
},

As we see in the above, all the storage is used by doc_values which is expected as the field is set as a tiem_series_metrics. No inverted_index exists.

Last, we check if the field is also queried as expected:

GET metrics-system.memory-default/_field_usage_stats

"system.memory.actual.used.pct": {
  "any": 3,
  "inverted_index": {
    "terms": 0,
    "postings": 0,
    "term_frequencies": 0,
    "positions": 0,
    "offsets": 0,
    "payloads": 0,
    "proximity": 0
  },
  "stored_fields": 0,
  "doc_values": 3,
  "points": 0,
  "norms": 0,
  "term_vectors": 0,
  "knn_vectors": 0
}

The result is good news, only queries are run on doc_values and queries are run.

Next steps

If we look at other fields in metrics-system.memory-default, we can find fields which are mapped and ingested, but not used in the queries / visualisation. The basic idea I have is that could be done for an integration when running a test:

• Load assets
• Run queries (dashboards, visualisation). API to extract it ([PoC] Dashboard queries API kibana#173416) from @drewdaemon could help here
• Compare results if all fields are used and mapped correctly. If not, potentially remove fields or change mapping properties

Links

• What if Elasticsearch could do this on the fly? Automatic tuning of mappings of data streams elasticsearch#87469