[Search] Search sessions benchmarking

[lukasolson]

Summary

In #73281, we opened an RFC for background sessions. As part of the review/comments, we identified a possibly-problematic issue with the current design. The design calls for a few server-side polling mechanisms:

1. For server-side implementations of background search, where each Kibana server needs to poll to see if a saved object for the given background session has been created (in order to update it with the current search)
2. For polling on async searches in Elasticsearch to get the progress information (and ultimately update the saved object representing the background session when all async searches complete) (we plan on using task manager for this)
3. While the end user is present, polling for partial results for the application implementing partial results

For (1), we don't plan to poll forever, but only until some threshold has been reached (5 mins has been proposed).

Particularly (1) and (2) may be problematic when there are many users executing multiple background searches/sessions. Since ll server-side processes must share the same event loop, adding these additional polling operations may affect other features such as alerting, task manager, etc. We need to come up with a baseline to compare against, then come up with some benchmarks to identify how problematic these polling operations are.

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[lukasolson]

/cc @kobelb, interested to hear how you've been benchmarking with Fleet and see if there is ultimately anything we can also do here.

[kobelb]

While benchmarking Fleet, we've been trying to maximize the number of Agents which can be connected to an instance of Kibana. To do so, we've created a replicable environment in GCP where we can simulate a number of Agents all connecting to the same Kibana instance. While these Agents are interacting with Kibana, we want to ensure that they're not blocking all other Kibana operations and fully consuming Kibana's available resources. To do so, we continually poll another Kibana endpoint and record its response latency as a health-check. If the health-check response latency crosses a threshold, we consider the load exerted by Fleet to be unacceptable. We're using a purpose-built Go application which exerts the load and monitors the health-check endpoint https://github.com/roncohen/fleet-loadtest. We also utilize metricbeat to monitor the Kibana instance during the load test to measure the CPU and memory consumption.

If I had to do it all over again, instead of using a purpose-built Go application to exert the load, I'd prefer to use a tool like Gatling or k6. I know that @dmlemeshko has done some load-testing of Kibana with Gatling and found it acceptable. I'm not sure whether or not he assessed k6 and has any opinions to provide 😄

What I'd recommend we do for Background sessions is benchmark how many concurrent searches Kibana can support prior to implementing background sessions and then after. We should be able to follow a similar process we took for Fleet where we figure out how to replicate the load using a tool like Gatling or k6, and incrementally ramp up the number of concurrent users until we cross the threshold for the health-check response latency.

It's potentially worth stating that we'll never be able to exactly replicate real-world usage with these types of load tests. The load which we'll exert in testing is likely a lot more uniform than what will happen in production, and that's tolerable.