[Fleet] Index patterns for integrations not showing up if any indices exist

[neptunian]

In #85627, implemented in #85370, changes were made to fix index patterns being correctly returned by the fields caps api by passing in an allowNoIndex property to the index pattern saved object. This fixed the case where no index patterns show up if no indices exist and use the index patterns stored in the saved object instead. However there is still the edge case where if an index DOES exist, but an integration with index pattern fields was installed and no data for that integration has arrived yet, those fields will not show up. Or if an index already exists that matches that index pattern for whatever reason. This is easy to see by going to the Fleet app. After setup, system fields should exist in Stack Management -> Index Patterns -> metrics-* because the integration was automatically installed, but no system fields exist because an index was already created during setup called metrics-endpoint.metadata_current_default and so the field caps api uses the fields in that index instead of the ones that exist in our metrics-* saved object.

We need to:

• decide whether or not we are okay with this behaviour. No error is thrown in any case.
• see what the differences are between creating the index pattern ourselves and letting field caps api return the index pattern fields based on existing indices and which serves our purpose better. If we always want the index pattern we create in the saved object to be returned perhaps we can have a new flag that always looks at that.

cc @mattkime
Related: #85370, #82223

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[neptunian]

@ruflin @mattkime I think the quickest fix for this is to have a flag that we pass to the API that always looks at the saved objects we create and not the indices. So we can potentially rename allowNoIndex to something else that could handle that whether or not indices exist.

[mattkime]

@neptunian I took a close look at this issue but its unclear to me how users are impacted by the current state. Its true that some fields aren't listed in index pattern management. Is it common for users to modify their index patterns before they have data? Does this problem display itself in other ways?

I'm very hesitant to work with a static field list as it removes flexibility thats present for any other index pattern.

[neptunian]

Here is an example where the user has installed the system package before ingesting any data and views a dashboard. The index pattern fields are not returned from the api, though they exist in the saved object:

I think the only other concern is what @ruflin mention to me that we are not sure what we might be losing out on by letting Kibana create the fields based on existing data. But this is something we need to compare and understand for ourselves. In the event we aren't missing out of anything and we're okay with the error above or can make a nicer error or remove it, we could possibly get rid of creating the index patterns ourselves in the long term.

[mattkime]

@neptunian - What about the screenshot is wrong? I see two fields displaying 'field not found' errors - not ideal. Is there anything else?

But this is something we need to compare and understand for ourselves.

Yes, I think this is the case. If there was anything that was being lost it would have been evident in the past if an index pattern had been refreshed.

[neptunian]

@neptunian - What about the screenshot is wrong? I see two fields displaying 'field not found' errors - not ideal. Is there anything else?

As far as I know, that is all.

[mattkime]

Thats something I'd like to see addressed. It may be possible for those visualizations to hide the error and simply display 'no data'. IF thats all thats needed, I think we could simplify other portions of how allowNoIndex is handled. Our strategy becomes about hiding errors instead of preserving fields.

[petrklapka]

Update for stakeholders: This issue is being actively discussed. It has been determined that there is no quick and easy fix. We are evaluating different holistic approaches and debating the merits and effort levels of three different options. Based on the complexity of all three of them, the current guidance is that the issue will not be fixed until after the 7.14 release. When a technical approach is agreed upon, we will post more information.