[Feature Request] Support a verbose/debugging param in search pipelines

[ohltyler]

Is your feature request related to a problem? Please describe

As the library of search pipeline processors continues to grow and become more complex, it can become increasingly difficult to know how data is passed around and transformed through the processors. An example is the introduction of ML inference processors which can have logic to transform arbitrary and complex model inputs/outputs.

Describe the solution you'd like

Having a way to debug and view the state of search processors (on request side and response side) would be helpful in discovering issues related to data transformation or any other intermediate failure. It could also be generally useful in being able to view the end-to-end pipeline execution and easily see how the request is transformed, executed, and how any response data is transformed. Additionally, this could be consumed and viewed on the flow framework frontend plugin, which is initially focused on the configuration and testing of ingest and search pipelines as users build out their complex use cases.

A few different implementation ideas:

1. Add a verbose parameter to a search request that contains a search pipeline, or standalone API, for returning the end-to-end breakdown of each processor's output - note this is already done today in ingest pipelines (see verbose param here)
2. Add some return_request parameter to a search request that contains a search pipeline, and return the finalized/transformed search request that was used to execute against an index. While this wouldn't be processor-level granularity, it could be a simple way to get some intermediate information.

Intuitively, I think Option 1 provides the most flexibility and simplest/straightforward design. It is also consistent with how ingest pipelines supports this idea.

Related component

Other

Describe alternatives you've considered

No response

Additional context

No response