[GS] Advanced search syntax to fine tune results returned

[alexfrancoeur]

Assuming we end up adding more granular search results (#72680, #74283) we'll likely want a way to narrow down the search even further. Adding some sort of simple syntax will likely be necessary to fine tune navigation. Some examples below:

• metrics: aws-host2
• cases: malware and cases:open
• dashboards: soc

I think this will only be necessary as we introduce new result providers to navigational search. We'll also need to make sure we get the UX / syntax right.

Improvements / follow-ups:

• add the filter suggestion option [GS] add tag:xxx or type:xxx filter suggestion option #83888
• should we translate the search syntax? [GS] internationalisation of the search syntax #83887
• add type alias support in type filter (canvas-workpad -> canvas / workpad)[GS] allow type 'alias' for search syntax #83889

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[elasticmachine]

Pinging @elastic/kibana-core-ui (Team:Core UI)

[pgayvallet]

Functionally, this makes sense imho. technically, we will need a definitive syntax before starting to implement a parser. Depending on the requested syntax features, this may be non-trivial.

[joshdover]

Depending on the requested syntax features, this may be non-trivial.

I vouch for using PEG.js to generate a parser/lexer from a formal grammar spec. We have prior art here in the expressions / interpreter code which uses the same library for parsing canvas expressions.

[ryankeairns]

We've been doing some work to line up all the GS requests thus far in anticipation of prioritizing work beyond 7.10. We'll obviously need your input as we have those discussions. Related, I've started a GH project to track ongoing work for this feature: https://github.com/elastic/kibana/projects/65

[pgayvallet]

the syntax

So, after thinking a little about it, maybe we should be using EUI's Query syntax here. The two arguments I got are:

• This is the only search syntax widely used on Kibana at the moment. Users would not have to learn a 'new' syntax (For example selecting a type to filter on in the SO management section updates the search field using this Query syntax)

• The lexer already exists. we would 'only' have to implement the parser and to use the parsed query from within the result providers (mostly the SO provider)

Note that this syntax is slightly different than the one suggested

• cases: malware and cases:open would be cases:(malware and open) (or, eventually cases:malware cases:open)
• dashboards: soc would be dashboards:soc (no space allowed between the field/value and the : separator)

the features / filters we can provide

FYI, The query documentation is available here: https://elastic.github.io/eui/#/forms/search-bar.

Technically, for the SO result provider, we have two options:

(The easy one) using the search, type, references searchField options of the find API.

This limits our filtering options to:

• search only for specific types (using the same search term) type:(dashboard OR visualization) my term or dashboard:"my term" (but NOT dashboard:myterm OR visualization:another-term)
• search only for objects assigned to the given tags (using the same search term) tag:(tag-1 OR tag-2) my term
• search only on specific fields - description:(my term)
• inclusions of the previous options, i.e type:(dashboard OR visualization) tag:(tag-1 OR tag-2) my term

(The hard one) convert the client-side ast to KQL

For more advanced search needs, the find SO api allow consumers to pass a KQL query to be used for the search/filtering. If we can convert from the Query AST to a KQL node, we could add more advanced features, such as searching for different terms depending on the type of object, i.e dashboard:my-term OR visualization:another-term.

However it's important to note that:

• converting a syntax to another syntax is a very challenging task. The work here would be significant
• EUI's Query only supports per-field filtering (field:value another-field:(value1 OR value2 ) my search term). It would not allow us to parse things such as dashboard:my-term OR visualization:another-term. For this solution, we would need to implement our own lexer.

what do we want exactly

So, before digging any further on the technical level, I would like to know what we expect (ideally short and mid term) from this feature.

Regarding the issue's description, it seems the only need would be to be able to filter on a specific type using mytype:myterm or mytype:(myterm OR anotherterm)

So my questions are:

• Would the EUI Query syntax be acceptable, or should we directly exclude this option and create our own lexer
• Apart from type:term, what other kind of filtering do we expect for a MVP of the feature

Side note

type:term syntax

the search syntax interpretation will be at the GS API level, meaning that when a user type mytype:(myterm OR otherterm) search term, the result providers will receive an object that could look like

{
   term: 'search term',
   raw: 'mytype:(myterm OR otherterm) search term',
   fields: {
    mytype: ['myterm', 'otherterm']
  }
}

I would really like to avoid the type:term (i.e dashboard:flight) syntax in possible, in favor of the type:dashboard flight one (unless this is the ONLY type of advanced filter we would provide, which I doubt)

The reasoning is that mixing filters such as type:term (dashboard:flight) with field:value (tag:my-tag) will introduce a lot of complexity for providers to interpret the query.

For example, as the applications result provider, I know that if I receive { type: 'dashboard'}, I must not return any result, as type would be a statically defined field in the official GS documentation. On the other hand, If I receive { dashboard: 'some term'}, this is a dynamic field that depends on the list of existing saved objects. The provider would not have any way to understand that the user is searching only for dashboard.

We could, eventually, process this type:search syntax in, and pass down a processed object to the providers. That way, dashboard:flight would be returned as

{
   term: 'flight',
   raw: 'dashboard:flight',
   fields: {
    type: ['dashboard']
  }
}

However, this transformation feels non natural. Also, What are we suppose to understand from a query that would be dashboard:flight another search term than flight?

Result providers implementation overhead

As already said, the search syntax interpretation will be at the GS API level, meaning that when a user type mytype:(myterm OR otherterm) search term, the result providers will receive a structured object resulting from the query parsing.

however, the actual usage of this object will be delegated at the result provider's level. Meaning that any additional result provider we would be adding (or any solution team would be adding) would have to also be able to support the filtering, which is additional (significant) work, to be integrated into the GS results.

How do we educate our users

Advanced search syntax are something that can be very hard for users to lean, or even to understand.

• How do we keep that simple enough
• How do we educate our users, and document the filtering possibilities of the navigational search
• How do we handle syntax errors when the user types something that is invalid for the lexer