[Feature Request] Fine grained control of (ingest|search) processors to install

[andrross]

Is your feature request related to a problem? Please describe

I have a deployment of OpenSearch that uses dedicated coordinator nodes. I need to be able to strictly control the work that happens on these nodes, and in particular I want to limit CPU-bound work. Thus far I have disabled the use of search and ingest pipelines by restricting traffic to the corresponding endpoints that create pipelines. I now have the requirement to enable some pipelines (specifically the pipelines defined by the neural-search plugin). However, all default ingest and search pipelines are defined in their corresponding modules (ingest-common and search-pipeline-common), which are installed by default with no way to opt out. Because these pipelines would introduce a new class of CPU-bound workload on the coordinator nodes, I do not want these pipelines to be enabled in my installation. OpenSearch provides no way to disable the registration of these pipelines.

Describe the solution you'd like

I would like a mechanism to not install all the default search and ingest processors. This could be implemented by a new setting (one each for search and ingest). It might work like the following:

• Unset (or a special token like "_all"): All default processors are installed. This is the behavior today.
• []: No default processors installed.
• ["foo", "bar"]: Only processors named "foo" and "bar" are installed.

Related component

Search Pipelines and Ingest Pipelines

Describe alternatives you've considered

Given that all the default processors are installed via modules, and my current requirement is to install none of them, then this could also be solved by a mechanism to opt-out of certain modules all together. I think this can be done today with an rm -rf modules/ingest-common command after installation, but this feels a bit like a hack. Also, opting out of the modules all together is less flexible than the setting mechanism proposed above.

[getsaurabh02]

Untriaged as part of the Search Community. We discussed high level approaches on providing a static control on allowed list of processors, which can then be further expanded to dynamic with access controls on who can add/update them