[Lens] meta fields are not supported in visualizations

[rajvi-patel-22]

Description

Below two SpringBoot dashboards contains Audit Events [Logs Spring Boot] panel which uses _id field:

1. [Spring Boot] Audit Events
2. [Spring Boot] Overview Dashboard

These visualizaions are working as expected in 8.4.0 and 8.5.0. But in Kibana 8.6.0 and later versions it shows below error.

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[elasticmachine]

Pinging @elastic/kibana-visualizations @elastic/kibana-visualizations-external (Team:Visualizations)

[stratoula]

@rajvi-elastic how was these visualizations been created? _id is not aggregateble so it doesn't appear on Lens field list 🤔

This is from 8.5

As seen on the Es documentation

The _id field is restricted from use in aggregations, sorting, and scripting.  In case sorting or aggregating on the _id field is required, it is advised to duplicate the content of the _id field into another field that has doc_values enabled.

[rajvi-patel-22]

@stratoula, In cloud instance it is showing _id field is not aggregateble. But when I checked in on-prem ES instance, _id field is aggregateble.

[stratoula]

Hmm, I dont know why, from the Lens side if the field is aggregatable it will be shown in the list and can be used in aggregations.

@elastic/kibana-data-discovery any thoughts here?

[lukasolson]

Hmm, this is definitely odd. Is there anything different in your mappings for the on-prem vs. cloud cluster?