[Search Pipelines] Base Benchmarks

[macohen]

Benchmarking is a good way to measure how changes to our code impact performance over time. This RFC should cover benchmarking for Search Pipelines and processors that will be included in the release.

Use the OpenSearch Benchmarking Tools to benchmark a very simple search (benchmark pipelines, not search):

• using the _search endpoint to baseline
• using a search pipeline with no processors (how much overhead for search pipelines vs _search alone)
• for each processor, benchmark a pipeline with only that processor in use. (how much overhead for each processor)
• for all processors available in core, benchmark a pipeline with all of them in use (this may be a little silly if we have 12 reranking processors), but if we think of it as eXtreme Search Pipelining it will sound cooler.
• encourage, but do not require processors that are not part of the OpenSearch Project release to benchmark their processors, as well.

What can we do in the short term?
Do these make sense for benchmarks? What else would we want to measure?
How do we set up the OpenSearch Benchmarking tools for this?

[msfroh]

using a search pipeline with no processors (how much overhead for search pipelines vs _search alone)

These are identical -- they follow the same code path. No search pipeline uses the "no-op" pipeline (which is a pipeline without processors).

[noCharger]

Search Pipelines and Processors Benchmarking Plan

Goal

The primary goal of this benchmark is to evaluate the performance impact of the various search pipelines and processors available in the OpenSearch Project release. We will measure this by comparing the performance of the _search endpoint with pipelines and processors of various complexities.

General Assumptions

• All tests are performed in a controlled and isolated environment with the same hardware specifications, avoiding external factors that may affect the benchmark results.
• Tests are conducted using the same set of data to maintain consistency.
• The performance measurements focus on the time taken to process the queries, but other metrics such as CPU usage and memory consumption can also be considered.

Benchmarks

The following table outlines the tests we plan to conduct, their features, and any notes related to them:

Feature	Test Case	Notes
Baseline	Use the _search endpoint alone without any pipelines or processors.	This will provide the baseline for comparison.
Pipeline without Processors	Use a search pipeline without any processors.	Measure the overhead for no-op pipeline alone.
Single Processor	For each processor, create a pipeline with only that processor in use. Also query with the same processor via _search request as ad-hoc.	Measure the overhead for each processor.
All Core Processors	Create a pipeline with all available processors in use. Also query with the same processor via _search request as ad-hoc.	An extreme test to measure overall impact.

Future Work

While this benchmarking plan provides a good starting point, we will need to iterate and refine it based on our findings. The results of these tests will help us understand how different search pipelines and processors impact the overall performance of our system and guide us in future optimization efforts.

[noCharger]

For 2.9 we will have the data around. For 2.10, we are planning to have the benchmark dashboard integrated.

[noCharger]

Optional:

• 1. Support search pipeline CRUD in python client [FEATURE] Add support for search pipeline APIs opensearch-py#474

There are two main effort:

• 1. Add new runner in OSB Add support for create search pipeline opensearch-benchmark#364
• 2. Add new operations and test procedures in workload repo Add search pipeline test procedures opensearch-benchmark-workloads#101

[noCharger]

Close this issue since all tasks are merged