Apply context filters when retrieving list of fields in application

[jimczi]

In elastic/elasticsearch#56195 we added the ability to filter the field capabilities output with an index filter.
The idea is that _field_caps can dynamically retrieve a list of fields that are relevant for the context.
If a user in Discover has a range filtering the last 3 days, we should restrict the list of available fields for suggestions to the ones that appear in the range.
Our Observability solution also uses constant_keyword to differentiate data streams so applying the current filters to the _field_caps call could limit the fields to a specific metricset like cpu for instance.

This change is important for our Solutions in order to have a way to limit the number of fields in large index patterns when the context is narrowed (by a range filter or a filter on the dataset).
Today the list of fields of an Index Pattern is retrieved once on each app through field_caps without taking the context into account.
We should make it even more dynamic and apply the context of the apps more eagerly when possible (changing the range filter should update the list of available fields).

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServices)

[mattkime]

I think the next step for this would be to design a specific UX that would benefit from this enhancement. Which solution?

@ruflin You spoke up on the ES issue, I'm curious if you have any ideas.

[ruflin]

++ on what @jimczi described. On my end I had initially mainly Dashboards in mind that a dashboard could define its prefilters for fieldcaps and looking at a mysql dashboard, only the related mysql fields are suggested.

On the solution side a few ideas:

• APM: APM has a specific set of indices. In the future these are across logs, metrics, traces but can be prefiltered to only show the releveant ones. @sqren
• Endpoint UI: Same as API, which indices to be used is known in advance. @kevinlog
• Metrics UI: I expect his is also a predefined subset to filter on. @jasonrhodes

[sorenlouv]

If a user in Discover has a range filtering the last 3 days, we should restrict the list of available fields for suggestions to the ones that appear in the range.

Yes! I'd love that. We could really need that in Observability for the search bar that currently suggests many irrelevant fields. I think that would also mostly solve #94879
cc @andrewvc

[kevinlog]

cc @peluja1012 @XavierM - this could be useful for future iterations of the global search bar in the Security app. We could restrict some field suggestions.

@sqren

Yes! I'd love that. We could really need that in Observability for the search bar that currently suggests many irrelevant fields. I think that would also mostly solve #94879

This would also be helpful for us in the Security UI, however I think we'd still like to "map" the suggestions to a more human readable format instead of always showing the fields as the appear in the document. What @andrewvc mentions here: #94879 (comment) is still relevant and should also be solved.

[mattkime]

Could a solutions team drive the planning for this effort? This needs a thorough UX story - How does the user set the filter? (either directly or indirectly via an existing control) Do these filters persist between Kibana apps? Perhaps there are other questions we should be asking but those are the first that come to mind.

From there I'd need to figure out what is needed from the index patterns api.

[andrewvc]

++ to @kevinlog , it doesn't hit the usability threshold we need.

While I think this proposal is useful it doesn't solve #94879 , though it might be a step in the right direction. What that issue calls for is a human curated list of fields that are useful. If Elasticsearch mappings had the ability to tag fields as 'friendly' or 'high priority', that would be more helpful, though I'm not convinced that's the right approach. That would look something like what is below:

{
  "monitor.name": {
    type: "text",
    friendly_name: "Monitor Name",
    description: "The monitor's human readable name"
    priority: true // definitely needs a better name, 'suggested?'
  }
}

The point is we just want to show the most frequently used fields. In any given solution there are lots of internal fields that are used for various purposes but rarely queried by users. We should not suggest these.

The real question is where do we do this work. Do we do it in ES? In Kibana? If Kibana, do we do it in an Index Pattern? Something else on top of one?

My $0.02 is that it's better to do it in Kibana and bundle a field list with each solution or package. I'd just make this a feature of the Kuery Bar UI component.

I'm on vacation starting next week lasting through mid-month, but I'd be glad to move this forward when I return, though I won't have time before this.

[mattkime]

@andrewvc

The point is we just want to show the most frequently used fields.

Index patterns already has this via the field 'count' property. Discover is the only app that makes use of it but it certainly could be expanded. If a particular solution creates index patterns, count values for the various fields could be preset. In the UI, this value is labeled 'popularity'

[andrewvc]

@mattkime I realize the word 'frequently' did not convey my intent accurately. I meant it to mean the most useful fields, not the ones that were literally the most accessed.

Let's think about users who don't know every field in the Uptime or APM schemas. They don't want a field named monitor.duration.us, and then to query by microseconds, they just want "Monitor Duration", in seconds or ms. They don't even want to see most of the fields we have, many of which are esoteric, or best for internal use.

This list needs to be manually curated, not algorithmically generated. This whole problem is analogous to the situation between the new Exploratory View and Lens. Lens is a power tool with full access to the schema, and assumes the user is comfortable with, or willing to learn the under the hood schema. The reason the exploratory view is so powerful is it presents users with commonly used fields exclusively, and doesn't use complex schema dot notation, rather favoring friendly names, and only showing a small number of the most important fields.