[Discuss] Performance benchmarking improvements for Opensearch

[ankitkala]

Currently we have a very basic performance test suite(link) where we execute a single workload nyc_taxis on a single node cluster and capture the metrics. I wanted to open a discussion for process improvements in benchmarking Opensearch(periodically as well as during every release). This would help in a more through benchmarking and ensuring that we don't miss out on any regression.

Listing down few high level improvements that i can think of. Feel free to add more test scenarios.

1. Testing different cluster configurations
We should also cover different cluster configurations(multi-node clusters, with/without replicas(logical/physical), Multi-AZ configurations, Instance types varying compute, memory and storage(EBS/SSD).

2. Testing with different workloads
Existing list of workloads are mentioned here.
We should add different types of workload to simulate different traffic types like:

• geonames for structured data.
• pmc for full text search.
• nested for nested documents.

Apart from the existing workloads, we need workloads with higher volume of data(highest is nyc_taxis with 75 GB approx.). Here is an existing issue on Opensearch-benchmark for the same. Workloads like these would definitely help benchmarking larger clusters (like 100 nodes!!) which reflect real workload of biggest consumers of Opensearch.

3. Benchmarking other usecase(core or plugins)
Apart from search and indexing, we also need benchmarks for other features which are present in core or external plugins. Few examples are:

• Snapshots.
• Reindexing.
• Security plugin.
• Cross cluster search/replication.
• Remote reindex.
• Async search.
• SQL.
• Index management.
• Segment Replication.
• Remote store.
• Pluggable Translog.

[ankitkala]

@dblock @muralikpbhat

[dblock]

👍

[reta]

It would be great to have the details of the benchmark runs to be exposeable (publicly) and comparable so to spot obvious regressions easily (inspired by [1] and [2]).

[1] https://elasticsearch-benchmarks.elastic.co/
[2] https://home.apache.org/~mikemccand/lucenebench/

[ankitkala]

Agreed. Not sure if opensearch-benchmark team already has something like this on their roadmap.

cc: @kotwanikunal @travisbenedict

[ankitkala]

Another aspect for focus can be ease of doing performance tests for developers. Current scripts for perf requires provide the bundle manifest which contains the link to the artifacts. Support for perf tests using locally built artifacts(OS and plugins) would be super helpful.

[CEHENKLE]

@bbarani Can you comment? Thanks!

[travisbenedict]

I think testing with more of the existing workloads as suggested in point 2 of the original post would yield the most useful information and require very little effort to implement. We just might need to scale our existing infrastructure to handle more concurrent tests.

[kotwanikunal]

I wanted to summarize some details around the current performance testing framework and also identify the gaps that currently exist. This might re-iterate on the points mentioned as a part of the issue but will help layout the picture for the framework in place.

Current state of the world

We have nightly performance tests that -

• Run using a single node cluster
• Are executed using OpenSearch benchmark infrastructure
• Execute the latest build for the defined versions using the generated distribution
• Benchmarked using the nyc_taxis dataset
• Publish results to a S3 bucket and to an internal-only OpenSearch cluster for visualization

Gaps and issues

Apart from the issues mentioned above, there are a couple of more gaps -

• Observability and visibility
  • The tests currently publish raw results into a S3 bucket which are not discoverable and accessible by the community
  • The results need to be manually tracked for inconsistencies and anomalies
  • Ideally these results should be published to a public OpenSearch instance allowing the community to view the results and utilize the alerting plugin to notify of any inconsistencies for the nightly tests
• Feature based performance testing / Custom branch performance testing
  • Currently only the distributions from the build process are capable of performance testing
  • As a part of feature development, performance tests become a necessity to gauge improvements and/or side effects of the changes before merging them into main
  • The decoupled nature of testing and build is the correct approach to follow, but additionally a build job which can build for branches will help with this requirement.

[reta]

@kotwanikunal thanks for the summary, I would include into "Gaps and issues":

• Run benchmarks for all available workloads

[ankitkala]

The tests currently publish raw results into a S3 bucket which are not discoverable and accessible by the community

Agreed. Let's follow up with opensearch-benchmark whether this can be included as part of their roadmap. @travisbenedict

The results need to be manually tracked for inconsistencies and anomalies

Correct. There are already APIs for comparison that can be leveraged.