Pinot fanout constant queries to all servers which could cause unexpected DNS query spikes

[pengding-stripe]

For constant queries using v2 engine, Pinot randomly assign a broker and the broker forward the query to all servers to process. This could potentially cause cluster-wide inter-server communications and result in issues like burst of DNS queries. We saw this problem for some MSE queries with bad filters that filtered out all data and made the queries constant requests.

I understand that these are still valid query so Pinot randomly pick a broker instead of fail them, but Pinot doesn't necessarily need to fan out the query to all servers. Ideally, these queries could be fan out to any single server to process.

[ankitsultana]

@pengding-stripe : the new MSE Query Optimizer is capable of handling this if you set useBrokerPruning=true in the query options: #15959

The new query optimizer uses the servers selected by the left most table scan by default. If you set useBrokerPruning, then it will also prune segments and servers.

We'll be adding documentation for this feature soon and it will be made GA as part of the 1.4 release.

But for reference, can you share the queries you are running?

[pengding-stripe]

thanks @ankitsultana, I feel this option should be default to true for this case? Our queries are sent from the user, it's hard for us to validate all of them and users could make mistakes. Below is an example of query sent by user, I only included the problematic part though

...
    SELECT xxx
    FROM "xxx"
    WHERE "ts_bucket" >= "FromDateTime"('2025-04-02 05:00:00', 'yyyy-MM-dd HH:mm:ss')
      AND "ts_bucket" < "FromDateTime"('2025-03-28 04:59:59', 'yyyy-MM-dd HH:mm:ss')
      AND "max_received_at" < CAST(
        "FromDateTime"('2025-05-19 19:46:59', 'yyyy-MM-dd HH:mm:ss') AS BIGINT
      )
    GROUP BY 1
    ORDER BY 1 FETCH NEXT 10000 ROWS ONLY
  )

[Jackie-Jiang]

@ankitsultana I'm not sure if this is the same thing. The problem right now is that no table is queried because the predicate is evaluated to false in the planner. Ideally we can directly execute the query and return from the broker

[ankitsultana]

With #16221 , v2 query optimizer will solve these queries in the broker itself. fyi: @Jackie-Jiang