[GS] SavedObject results provider

[pgayvallet]

This issue is to discuss the concrete implementation of the SavedObject result provider needed for the GlobalSearch MVP: #58049

Related issues:

• GS meta issue: [Meta] Global Search #57576
• GS plugin API: Generic UI search API #61657
• GS API RFC: [RFC] Global search API #64284

From #58049, the types of SO we would need to return would be:

• Dashboards
• Visualizations
• Saved searches
• Canvas workpads
• Maps
• Graphs
• ML jobs (future)
• SIEM Timelines

Questions:

How exactly will we perform the query / which fields should we search by

It's still unclear which fields we should search by. The base SavedObject type do not even have a title field in the base implementation, and I'm not sure all types actually declare one in their mappings.

Second point, when calling for example gs.find('dash'), should we search on the type field and return all dashboard objects? The type field being a keyword, is this doable?

Is our current find API sufficient, or will we need to create a new one

Directly depends on the first point. I feel like we will need to add a new API to answer the GS needs, even if ideally this would be avoided.

Is it technically possible to retrieve all SO in a single query

For performances, it would be best if we could retrieve all types of saved objects in a single query, instead of performing x (= nb types) queries. Would this be a possibility?

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[elasticmachine]

Pinging @elastic/kibana-core-ui (Team:Core UI)

[pgayvallet]

CC to @rudolf @joshdover @ryankeairns @myasonik @kobelb

[rudolf]

How exactly will we perform the query / which fields should we search by
It's still unclear which fields we should search by. The base SavedObject type do not even have a title field in the base implementation, and I'm not sure all types actually declare one in their mappings.

Is our current find API sufficient, or will we need to create a new one
Directly depends on the first point. I feel like we will need to add a new API to answer the GS needs, even if ideally this would be avoided.

Is it technically possible to retrieve all SO in a single query
For performances, it would be best if we could retrieve all types of saved objects in a single query, instead of performing x (= nb types) queries. Would this be a possibility?

I don't think it makes sense for all saved objects to have a title, so it feels like it's better to keep this in the type attributes than to add a title field to the root property of all saved objects.

I think we should start with the Saved Objects Management plugin's search abilities as a baseline. It allows plugins to specify a defaultSearchField and searches across all types on these search fields using the SavedObjectsClient#find API:

kibana/src/plugins/saved_objects_management/server/routes/find.ts

Lines 66 to 77 in a7508b8

	importAndExportableTypes.forEach(type => {
	const searchField = managementService.getDefaultSearchField(type);
	if (searchField) {
	searchFields.add(searchField);
	}
	});
	const findResponse = await client.find<any>({
	...req.query,
	fields: undefined,
	searchFields: [...searchFields],
	});

I know of #42316, but other than that is this existing search functionality sufficient for searching across Saved Objects or do we need something more powerful?

More generally, SavedObjectsClient#find (and the implementation in SavedObjectsRepository) allows for searching across multiple types but only in a single namespace. I'm assuming as a first step we'll only search in a user's current namespace, but I it might be useful to know that there's a dashboard with a very high score that matches in another namespace that you have access to instead of having to manually switch between all my spaces to try to locate it. But if we need to support this we could enumerate the user's spaces and construct a find call for each space.

Second point, when calling for example gs.find('dash'), should we search on the type field and return all dashboard objects? The type field being a keyword, is this doable?

To let ES search on it we will have to change the type field mappings which will trigger a migration. Since we know all the saved object types before hand we could do a client-side search and if a type matches search on the full type name. This makes it harder to combine search queries like "finance dashboard" which we might want to give a high score because it matches the type and title of {title: 'finance', type: 'dashboard'}.

[kobelb]

I agree that starting with the way that saved-objects management does search as a baseline is reasonable. It definitely leaves something to be desired, but it's tolerable.

The SavedObjectsClient#find does support KQL, which gives us more flexibility. It seems like we could allow every saved-object type to construct their own KQL statement for their own type, which is then OR'ed together. I honestly haven't tried this, so there might be some limitations that I'm glossing over.

Restricting the searches to the current namespace for the initial implementation is completely fine, we previously discussed that limitation and everyone seemed onboard. I believe that @legrego is looking into how we'd relax this restriction for another use-case.

You can still do searches on keyword fields in Elasticsearch, they just don't go through analysis. This means that you generally need an exact match, which isn't great. Adding a text field would greatly improve this behavior.

[legrego]

Restricting the searches to the current namespace for the initial implementation is completely fine, we previously discussed that limitation and everyone seemed onboard. I believe that @legrego is looking into how we'd relax this restriction for another use-case.

Yep, I'm looking into this whenever I get free cycles: partially for GS, and partially for the IM/Fleet project. I have a very rough POC working, but getting it tested and production ready is another task altogether

[pgayvallet]

I think we should start with the Saved Objects Management plugin's search abilities as a baseline. It allows plugins to specify a defaultSearchField and searches across all types on these search fields using the SavedObjectsClient#find API

That seems like a good idea. Either leveraging this API directly, or construct our own based on the existing defaultSearchField property (that is even directly exposed by core)

More generally, SavedObjectsClient#find (and the implementation in SavedObjectsRepository) allows for searching across multiple types but only in a single namespace. I'm assuming as a first step we'll only search in a user's current namespace

Yea, as @kobelb already said, this is totally fine for V1.

To let ES search on it we will have to change the type field mappings which will trigger a migration. Since we know all the saved object types before hand we could do a client-side search and if a type matches search on the full type name
You can still do searches on keyword fields in Elasticsearch, they just don't go through analysis. This means that you generally need an exact match, which isn't great.

Yea it's not great. @rudolf do you see any impact on changing the type field mapping appart from the triggered migration?

@ryankeairns functionally, can we get you opinion on:

when calling for example gs.find('dash'), should we search on the type field and return all dashboard objects?

[rudolf]

Yea it's not great. @rudolf do you see any impact on changing the type field mapping appart from the triggered migration?

I don't think it would cause any issues. It's not optimal for performance but we can use a multi field to index it as both keyword and text if that's a problem. We should probably do a benchmark with 100K objects so that we don't have do to another migration if this does end up impacting performance when Saved Objects support much more documents.

[kobelb]

I think we should use a multi-field for the type. Changing it to just be text will cause the analyzer to tokenize strings like "something-awesome" into two terms: "something" and "awesome". This will then make it difficult to write queries to only return saved-objects with the type of "something-awesome".