[O11y][SpringBoot] Update `_id` field in SpringBoot dashboard

[rajvi-patel-22]

Description

Below two SpringBoot dashboards contains Audit Events [Logs Spring Boot] panel which uses _id field:

1. [Spring Boot] Audit Events
2. [Spring Boot] Overview Dashboard

These visualizaions are working as expected in 8.4.0 and 8.5.0. But in Kibana 8.6.0 and later versions it shows below error.

According to documention _id fields are not supported in aggregations:

The _id field is restricted from use in aggregations, sorting, and scripting. In case sorting or aggregating on the _id field is required, it is advised to duplicate the content of the _id field into another field that has doc_values enabled.

Related Issues

• [Lens] meta fields are not supported in visualizations kibana#155344

[rajvi-patel-22]

There are two possible solutions to resolve this issue:

Solution 1:

As mentioned in the documentation of _id, the content of _id field can be copied in another field using ingest pipeline and the new field can be used in visualizations.

The _id field is restricted from use in aggregations, sorting, and scripting. In case sorting or aggregating on the _id field is required, it is advised to duplicate the content of the _id field into another field that has doc_values enabled.

Solution 2 [Recommended]:

The current visualizations have below limitations:

• It can only show maximum 10000 events [Reference]
• Search visualization has better performance than lens table visualization. [Reference]

The panels are showing list of audit events. Hence, the saved search can be used to display all the events. The only drawback in creating search visualization is column name. In search panels, column names are not configurable.

Please find below screenshots of originial visualization and suggested panel:

Original visualization:

Suggested search panel:

[rajvi-patel-22]

@agithomas, The possible solutions of meta data field (_id) issue are mentioned in the above comment. Please let us know how we should proceed ahead.

[agithomas]

@rajvi-elastic , i think unable to retain the column name might impact the customer experience and as well add to the confusion.

So, my recommendation is to go with the Solution 1, considering it is less disruptive.
Are there any specific reasons because of which Solution 1 might not be ideal?

[rajvi-patel-22]

Yes @agithomas. There are some drawbacks of solution 1:

1. The panel will show limited events. [Reference]
   • 1000 events till v8.3.0 (As SpringBoot is migrated to 8.1.0, it could only show 1000 events)
   • 10000 events from v8.4.0 to v8.7.0
2. Search visualization (Solution 2) has better performance than lens table visualization. [Reference]

[agithomas]

The panel will show limited events. [Reference]

• 1000 events till v8.3.0 (As SpringBoot is migrated to 8.1.0, it could only show 1000 events)
• 10000 events from v8.4.0 to v8.7.0

I am not sure, if the usecase is practical enough when a user will go through more than 1000 records in a dashboard panel .

@SubhrataK , thoughts?

[SubhrataK]

@rajvi-elastic I agree with Agi, does not seem like a valid use case to me. You can proceed with Solution 1.

[rajvi-patel-22]

@SubhrataK Okay! We will create one custom field spring_boot.audit_events.document_id and copy the value of _id field into spring_boot.audit_events.document_id which can be used in visualization.

Note: When user will upgrade to the newer version, spring_boot.audit_events.document_id field will be added only to the new incoming documents. The data that is already present in indices will not get updated. Hence, the Audit Events [Logs Spring Boot] visualization will show documents which have spring_boot.audit_events.document_id field present (documents that have been ingested after upgrading to the latest version).

[agithomas]

@SubhrataK Okay! We will create one custom field spring_boot.audit_events.document_id and copy the value of _id field into spring_boot.audit_events.document_id which can be used in visualization.

Note: When user will upgrade to the newer version, spring_boot.audit_events.document_id field will be added only to the new incoming documents. The data that is already present in indices will not get updated. Hence, the Audit Events [Logs Spring Boot] visualization will show documents which have spring_boot.audit_events.document_id field present (documents that have been ingested after upgrading to the latest version).

cc @lalit-satapathy .