Generic UI search API

[joshdover]

To power Global Search (#57576) and future features like ChatOps, we should provide a general framework for searching for various objects, apps, and other locations in the UI that a user may want to find quickly.

This framework should allow plugins to provide different "data sources" of which may be backed by Elasticsearch indices or other items (eg. application links).

The search framework should be able to take a search inputs, call each data source, and aggregate results into something sensible.

Tasks

• GS API RFC - [RFC] Global search API #64284
• GS API implementation - Add globalSearch x-pack plugin #66293
• implement SavedObject result provider - [GS] SavedObject results provider #65222
• implement Application result provider - [GS] Application results provider #65633

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[elasticmachine]

Pinging @elastic/kibana-core-ui (Team:Core UI)

[pgayvallet]

Regarding global architecture (actual required savedObjects dataSource(s) implementation will have it's own discussion)

dataSource registration and architecture

This framework should allow plugins to provide different "data sources" of which may be backed by Elasticsearch indices or other items (eg. application links).

In an ideal world, datasources would only be registered on the server-side, and the aggregation logic would also be contained on the server. core would then expose both a server-side contract API to be able to use this 'global search' from the server part of plugins, and an endpoint (with an associated client) to be able to use it from the public part of plugins.

However, we already know that some searchable objects (applications) are currently only available from the client-side (see #57576 (comment))

The first consequence is that we will need to allow datasources to be registered from the client-side.

A corollary from this first statement is that searching from the server-side will not provide all results, as it would not be possible to 'query' the client-side for client-side-registered sources.

We got an architectural decision to make here:

Option A: only allow to register dataSources from the client-side

Data providers would only be able to register dataSources from the client-side. If the datasource requires to perform calls to the server-side to retrieve their results, it will be considered an implementation detail.

Pros:

• reduce API exposure
• more consistent, every datasources are registered the same way
• results aggregation logic only performed one, on the client-side

Cons:

• more work for dataSource implementations, as it would requires to create the endpoint on the server and call it from the dataSource
• Even if chatOps has been excluded from the requirements from now, it will make it way harder to reintegrate it later

Option B: allow to register dataSources from both client-side and server-side

As most of the currently identified datasources (AFAIK, all of them except the applications one, see #58049) needs to perform server calls anyway, it would be possible to allow plugins to register dataSources from either the client-side or the server-side.

Pros:

• less work for dataSource providers, can just register it from the server-side
• would be able to have a server-side GS API with 'degraded' search results (without the client-side datasources' results)
• (follow-up of previous point) probably more 'future needs' proof, as we would be able to expose a server-side API for GS (i.e for chatOps), even if degraded.

Cons:

• Implementation more complex for core. We'll have to handle results aggregation twice: once on the server to fetch all server-side datasources, then on the client-side to merge the server results and the client-side dataSources results

I'm leaning to option B, but it's not a strong opinion, and would really like to have the other's vision on that.

POC interface for the dataSource providers

This is a base, naive contract to open the discussion, please comment on that.

// are fields missing here?
interface GlobalSearchResult {
  // the main text/title to display in search results UI
  title: string;
  // the type of result. also displayed as line2 in search result UI
  // I.E `dashboard`, `application`
  type: string;
  // EUI icon name to display for the search result.
  // Optional, will display a default (or no icon?) if not provided
  icon?: string;
  // The url to navigate to this result
  url: string;
  // optional, priority/ordering of the result
  order?: number;
}

interface GlobalSearchDataSourceResponse {
  success: boolean;
  // only if success == true
  results?: GlobalSearchResult[];
  // only if success == false
  error?: string;
}

// type instead of just a `(term: string, options: {}): Promise<GlobalSearchDataSourceResponse>` function
// to allow potential addition to the `dataSource` contract in the future.  
type GlobalSearchDataSource = {
  search(term: string, options: {}): Promise<GlobalSearchDataSourceResponse>;
};

// would be used like
coreSetup.globalSearch.addDataSource(myDataSource);

Remarks / Questions:

• How to handle redirecting to a result's associated page.

In current POC implem, I just added a GlobalSearchResult.url string that is supposed to be the url to redirect to. But it's definitly not enough:

In stage 1 of GS, we will only have objects contained in the kibana instance we are searching for, however in next stage we will have to display result from other clusters' kibana instances, and therefor be able to redirect to an url outside of current kibana's domain.

Navigating to an internal object should probably be performed using application.navigateTo, meaning we would need to have a { application: string, path?: string } object in the search result to be able to navigate to it.
Navigating to an object outside of the kibana app will need a 'real' absolute url to the object, meaning we will need an url: string for that.

Would something like

interface GlobalSearchResult {
  // ...
  navigateTo: { absUrl: string } | { application: string; path?: string };
}

suit the need?

• Results scoring / ordering

In current POC, I just added a GlobalSearchResult.order? field to allow dataSources to return an order/priority for their results. It's probably not good enough.

As we are going to fetch results from multiple dataSources, I'm not really sure how to properly introduce a global/centralised scoring mechanism, as it would relies on the underlying dataSources to score their results, not to the GS service/aggregator, which mean distinct dataSources would have different scoring mechanism, therefor making them biased and useless.

I don't see any good option on that problem, so any insight is welcome.

• Do you see anything else?

Last point, should I create a RFC for that issue? If so, should I do it right now or after we find answers to these problematics?

ping @elastic/kibana-core-ui

[ryankeairns]

Thanks @pgayvallet , I'll bring this up at our sync today.

@kobelb your input would be appreciated here as well!

[myasonik]

dataSource registration and architecture

Don't really have an opinion here but do have a hazy recollection from an early meeting with @stacey-gammon that a client-side only implementation ended up being the leading idea. Stacey, do you remember this any better than I do? Have any thoughts on @pgayvallet's proposal? 👆

POC interface for the dataSource providers

How to handle redirecting to a result's associated page?

navigateTo: { absUrl: string } | { application: string; path?: string }; seems good 👍 Haven't had a look into it but would this, in the future, let us switch spaces as well or would we need more?

Results scoring / ordering

Having a "global" level of scoring would be cool but I'm not sure that it's necessary. I think it'd be totally reasonable if the results were grouped by their type so all that the aggregator would then need to do is order the types themselves. If we could do some sort of scoring on which type is most likely to be the one we want, that could be neat but I think also just having a predefined hierarchy would be fine.

Anything else?

1. Not sure how strict you're being with types, but would GlobalSearchResult.type be better as an enum/string literal union of known types?
2. How future looking do we want the GlobalSearchResult interface to be? Other things that we've at one point or another mentioned possible being displayed that would need to exist here could be: space; for certain types, author, last edited, created at.

[pgayvallet]

@myasonik Thanks for the input

Don't really have an opinion here but do have a hazy recollection from an early meeting with @stacey-gammon that a client-side only implementation ended up being the leading idea

TBH the hybrid server/client implementation is mostly driven by the need to able to use GS for external APIs such as ChatOps. I really don't know how much using GS outside of the UI is a real / realistic need and if the degraded search results would be acceptable. If it's not, my option A is probably the way to go. Not really sure who should have the last word on that...

Haven't had a look into it but would this, in the future, let us switch spaces as well or would we need more?

application.navigateTo stays on the same space. Allowing to display results from other spaces and then navigate to them would need an additional information.
The struct could probably be enhanced to add a space optional property then.

I see two issues here however:

• Not sure how to programmatically change space. @kobelb is there a programmatic API for that? Can we also navigate to like a specific url of a given space.
• (probably more important) ATM the GS service is supposed to live in core, meaning that we won't be able to access the space plugin API anyway, which is kinda blocker.

If we choose option A regarding dataSources (only allow to register them from the client-side), we no longer need the GlobalSearchResult type to be serializable, as instances will only live on the client side, and we could then let the datasource providers handle the navigation themselves.

navigateTo: { absUrl: string } | { application: string; path?: string } would become navigateTo: () => void

with a example implementation of navigateTo: () => application.navigateTo('dashboard', { path: 'my-dashboard'})

This would allow dataSources providers to have more control on how they should handle access to the result, while definitely breaking the possibility to have a server-side API counterpart. This is a high-impact decision.

Having a "global" level of scoring would be cool but I'm not sure that it's necessary. I think it'd be totally reasonable if the results were grouped by their type so all that the aggregator would then need to do is order the types themselves.

This would be very doable. The main question I see here is: is ordering by type sufficient? I'm thinking mostly of results from other spaces or even other kibana instances. Should dashboard type results from another cluster have the same priority than dashboard results from current kibana instance?

Maybe we should order by dataSource instead?

type GlobalSearchDataSource = {
  getDisplayPriority(): number;
  search(term: string, options: {}): Promise<GlobalSearchDataSourceResponse>;
};

wdyt?

Not sure how strict you're being with types, but would GlobalSearchResult.type be better as an enum/string literal union of known types?

I'm mixed on this one. The major downside I see with that is that 3rd party plugins would not be able to register dataSources fortypes not included in this static list defined in the kibana codebase, which seems kinda dealbreaker in term of extensibility?

How future looking do we want the GlobalSearchResult interface to be?

Ideally, any field added at a later time would be optional, to allow BWC with the existing dataSources. This is why we need to look a little further than stage 1 here.

[joshdover]

The first consequence is that we will need to allow datasources to be registered from the client-side.

A corollary from this first statement is that searching from the server-side will not provide all results, as it would not be possible to 'query' the client-side for client-side-registered sources.

@pgayvallet and I discussed this over Slack and we decided it's probably best to go with the hybrid approach, though preferring datasources to be registered on the server side when possible:

• Makes supporting ChatOps in the future easier since we won't have to move most data sources to the server.
• Allows us to avoid solving some of the hard problems right now: specifically how to return results for application links on the server-side. There are a few options to solving this, but we don't need to for GS and we can defer that work to whenever ChatOps becomes a priority.

Navigating to an internal object should probably be performed using application.navigateTo, meaning we would need to have a { application: string, path?: string } object in the search result to be able to navigate to it.
Navigating to an object outside of the kibana app will need a 'real' absolute url to the object, meaning we will need an url: string for that.

I think we may be able to get away with just using absolute URLs, while still supporting navigating to other spaces and doing in-app SPA navigations. This idea is contingent on being able to parse out the application id from the URL, but I think this algorithm would be quite simple:

• If all 3 of these criteria are true:
  • The domain of the URL (if present) matches the current domain
  • The pathname of the URL starts with the current basePath (eg. /mybasepath/s/my-space)
  • The pathname segment after the basePath matches an application route (eg. /app/<id>/)
  • Then: match the pathname segment to the corresponding application and do the SPA navigation to that application using the remaining pathname segment
• Otherwise, do a full page navigation (window.location.assign())

Why this should work:

• When navigating within the same space, the basePath will match since it already contains the space prefix. Core won't need to have any knowledge of how spaces works.
• When navigating to another space, the basePath will not match and we will do a full-page navigation. We need to do a full page nav anyways in order to reload the UiSettings and the user's capabilities for the new space. (at least with how spaces works currently).
• When navigating to another Kibana cluster, will obviously need to a do a full page refresh.

Is there any case I'm missing here?

If we choose option A regarding dataSources (only allow to register them from the client-side), we no longer need the GlobalSearchResult type to be serializable, as instances will only live on the client side, and we could then let the datasource providers handle the navigation themselves.

I think this creates a non-trivial amount of work later when we do want to support chatops. I think having some constraints on what results can do is a useful design decision for future requirements.

[stacey-gammon]

Thanks for tagging me @myasonik. Here are my thoughts:

dataSource registration and architecture
Option A: only allow to register dataSources from the client-side
Option B: allow to register dataSources from both client-side and server-side

I'm leaning towards option B to avoid BWC issues down the road and pave the way for chat ops.

Urls & storage

UrlGenerators are registered client side, and the plan for now is to keep them that way, but to have the links accessible on server side via a simple helper:

// On server side
createGotoLink(urlGenerator) =>
  `${getDomain}/${getBasePath}/goto/${urlGenerator.id}?state=${risonEncode(JSON.stringify(urlGenerator.state))}`

You wouldn't need that though until actually generating the links to give to an end user. The following code could exist on server or client:

DashboardPlugin{
  setup(core) {
    core.globalSearch.addFinderDestination({
        search(term: string, options: {}): Promise<GlobalSearchDataSourceResponse> => {
          const [coreStart, ] = await core.getStartServices();
          const dashboards = coreStart.savedObjectClient.find('dashboard', term);
          return dashboards.map(dashboard => {
             title: dashboard.title,
             score: dashboard.score  // Can you reuse the score returned from ES here instead of "order"?
             urlGenerator: {
               id: DASHBOARD_URL_GENERATOR,
               {
                 id: dashboard.id,
               }
             }
          })
        }
    });
  }
}

I need to think through this a little more, but I want to be sure we are thinking through backward compatibility here.

With the URL Generator service, what is backward compatible is the id + state. NOT the url returned from it. I just gave a presentation on persistable state registries. I need to think more about where URLS fit into this picture.

Like, what if someone created a plugin that supported pinning search hits from this service? The pinned hit would be stored in a saved object with a url string with no way to support migrations from the owner of that url. If you stored the URL generator state, there is a migration path.

Thinking through a couple options:

1. Instead of returning the url string, return the url generator id and the url generator state. Create the href links on the fly, but you will still have the data so if someone did want to create that feature above, you'd have the data there that would be safe to store in a saved object in a BWC way (there is a way for an author of a URL generator to migrate the state they support). Chat ops could do this conversion too. Downside is that users of our API don't have a real URL to expose, they'd need to query yet another service to get it.

2. Encourage every implementor to preserve backward compatibility of the URL they use. They could use the goto links we are thinking of creating for the Url Generators:

http://domain/goto/{id}?state={risonEncodedUrlGeneratorState}

^^ that doesn't exist yet but it should to support bwc server side url links, because the app will grab the correct generator, decode the state, and use the Url generator which will have any registered migrations.

3. Have an official stance that these links should not be persisted, they are only good for the current version of Kibana and are meant to be used for "on the fly" searches, not storage. Would make the "pin this search result" feature more difficult.

Order

Regarding order. Maybe instead of order there should be a concept of "match score"? Just a thought.

Terminology

It may be too late in the game to change the terminology of "Global Search", though I do think this is going to be confusing. We have search services inside the data plugin. Maybe like "Global finder services" or "global resource locator" (... or, universal resource locator? 😄 )

Even if you are sold on Global Search, I really think you should steer away from "data source" because it's already used by canvas "data sources" and used heavily inside the ingestion manager.

Some suggested alternatives:

• Resource
• Page
• FinderPage
• Destination? FinderDestination?
• FinderItem
• LandingLocation?
• I wouldn't use SearchSource either because we already use that name for saved searches and "search" is already confusing.

The one given is that every item must have a url destination, right?

[myasonik]

Navigation

I like @stacey-gammon's first two options much more than using absolute URL strings. Relying on a bunch of string manipulation to pull logic out of it just tends to spider with condionals over time so having the actual backing data feels much safer.

No preference between the first two options but I think taking the stance that we will not support persisting URLs will come back to bite us so I'd avoid thinking of URLs as single use actions.

Ordering results

[Ordering by type] would be very doable. The main question I see here is: is ordering by type sufficient? I'm thinking mostly of results from other spaces or even other kibana instances. Should dashboard type results from another cluster have the same priority than dashboard results from current kibana instance?

I've got no idea. Almost seems like a product question? But also one that we might need data on from users... Also seems like an answer that is likely to change over time and/or how big the customer is. How much flexibility could we bake into our ordering strategies?

Maybe we should order by dataSource instead?

I'm not sure I follow on how this is different... What falls into a single if not a single type?

Maybe instead of order there should be a concept of "match score"?

Wouldn't that not be possible across multiple ?

GlobalSearchResult.type type (string or enum)

The major downside I see with [an enum] is that 3rd party plugins would not be able to register dataSources fortypes not included in this static list defined in the kibana codebase, which seems kinda dealbreaker in term of extensibility?

So, as a string, how would sorting results work if we have a/a bunch of type(s) that we don't know?

Something that would need a bit more plumbing but we could allow plugins to register types which could then be mixed into the types we provide to have both a known set of types and what to do with them while also letting developers extend Kibana.

[stacey-gammon]

Something that would need a bit more plumbing but we could allow plugins to register types which could then be mixed into the types we provide to have both a known set of types and what to do with them while also letting developers extend Kibana.

We have this kind of thing in use already in quite a few places, but I'm actually thinking we should discourage its use. Check out #62815 for more details. Although I'll reiterate here because that issue doesn't really touch on the type mappings.

You can see how URL generators does this here: https://github.com/elastic/kibana/blob/drilldowns/examples/url_generators_examples/public/plugin.tsx#L31

If you can type check this though, you know the id. E.g. you are doing something like

// If you use type mappings, this will be automatically typed and ensure
// that the state is of the shape the `FOO` generator requires
 urlGenerator.get(FOO).createUrl({ fooBar: 'hi; })

Because otherwise you are doing something like this:, which you can't type check anyway:

// We can't type check this, just have to hope where-ever this persisted data came from is
// using the right state.
 urlGenerator.get(id).createUrl(state)

So, in the former case, what I talk about in that issue, is that we should discourage direct access with the plugin if you know, at runtime, the type you want, and instead, get it off the registrators plugin directly. e.g.

 // Correctly typed
  foo.getUrlGenerator().createUrl(state);

Then you don't need typescript mappings at all.