[multistage] run planner for routing on controller

[walterddr]

Description

when query broker for a specific query. there's no need to parse out the tenant info.
however when using the Pinot controller UI. controller needs to first detect all the tables accessed by the query, then find the common tenant of a broker to route the query to.

Status Quo

currently this is done via a util done in #10336. however this util has lots of drawback and overhead. b/c it is not running the actual planner.

Proposed solution

1. run the calcite planner on controller (QueryEnvironment.compileQuery)
2. and use RelOptUtils.findTables to find all tables accessed by this query.
3. use the same util currently available to find common broker tenant list.

[walterddr]

CC @tibrewalpratik17 and @abhioncbr

[tibrewalpratik17]

@walterddr Just curious will this not add an additional dependency of pinot-controller module on pinot-query-planner?And what will we pass as WorkerManager to the QueryEnvironment object as it looks specific to broker / server configs?

Also for 2. we can use RelToPlanNodeConverter.getTableNamesFromRelRoot(relRoot.rel) which internally uses RelOptUtils.findAllTableQualifiedNames(relRoot) at the moment. This is a part of pinot-query-planner module itself.

[tibrewalpratik17]

And what will we pass as WorkerManager to the QueryEnvironment object as it looks specific to broker / server configs?

On deeper look I see we don't need WorkerManager in the compileQuery path.
Seems doable to me.

[abhioncbr]

Sounds good to me. I can work on this if it's not urgent. Thanks.

[walterddr]

@walterddr Just curious will this not add an additional dependency of pinot-controller module on pinot-query-planner?And what will we pass as WorkerManager to the QueryEnvironment object as it looks specific to broker / server configs?

Controller is never designed for sustained QPS workload --> in fact each pending query will hog up a thread/cpu thus it can handle maximum no_of_core query at a time.
So I think it is fairly ok to add a bit of overhead to this exploratory API for the UI and for improved user-experience.

Also for 2. we can use RelToPlanNodeConverter.getTableNamesFromRelRoot(relRoot.rel) which internally uses RelOptUtils.findAllTableQualifiedNames(relRoot) at the moment. This is a part of pinot-query-planner module itself.

yes, that's what I meant, we dont need to configure WorkerManager as we are not assigning worker at the moment, we only want to do the planning part.

Sounds good to me. I can work on this if it's not urgent. Thanks.

Thank you @abhioncbr, sorry I was originally intended to @ankitsultana but yes this is not urgent. please feel free to pick this one up at your convenience. (assigned to you)

[tibrewalpratik17]

Hey @abhioncbr is it okay if i pick this up and get this assigned to me?

[abhioncbr]

Hey @abhioncbr is it okay if i pick this up and get this assigned to me?

Sure. Thanks