[Proposal] Query Resource Isolation in Pinot

[vvivekiyer]

Hi, sharing the design for Query Resource Isolation in Pinot - https://docs.google.com/document/d/1bUr_l4B8gJCiUOKw97HFB9EEWvTU0Fw70HuZ0jq7YVw/edit?tab=t.0#heading=h.y9hbiahmee08

Problem Statement:
The noisy-neighbor problem is a serious limitation in dedicated tenant and shared tenant setups. In shared tenant setups, changes to query patterns/QPS/etc in one table can negatively impact the queries of other tables, increasing the blast-radius for incidents. In dedicated tenant use-cases, queries from a single use-case/upstream caller can impact other callers sharing the table.

Given a steady state workload being run on a Pinot cluster, Pinot should be able to protect itself and maintain the SLOs (availability and latency) and degrade gracefully to handle scenarios like unexpected changes in cost of workloads which can be a function of:

• New queries being ramped
• Changes to existing queries/query patterns
• New data characteristics changing query selectivity
• QPS spikes

Various versions of this problem have been discussed in the past. For example in #8618, #5627

High Level Proposal

Through this effort, table owners can define query workloads and attach enforcementProfiles to each workload. Pinot will provide read-path isolation for these query workloads. This will allow Pinot maintainers to provide better SLO guarantees and improve HW utilization.

For more details, please refer to the doc.

cc: @siddharthteotia @Jackie-Jiang @mayankshriv @mcvsubbu @ankitsultana @jasperjiaguo

[vvivekiyer]

We are starting with the execution of this effort. Gentle ping here to check if there are any suggestions/feedback you'd like us to incorporate. cc: @Jackie-Jiang @ankitsultana @mayankshriv @vrajat

[ankitsultana]

@vvivekiyer : overall lgtm.

Had a question though: how does this sit with the binary scheduler feature you folks shipped last year?

I guess the scope of this issue (#14551) is bigger, would there be a way where we could just use the features provided by this issue to get results that are similar to what we would get from the Binary Workload scheduler?

[vvivekiyer]

@ankitsultana Yes, this would be a bigger scope. We intend to create a scheduler as well for this but it will come after the MVP implementation.

We are currently benchmarking the PoC code. Once we have the results, we should be able to start creating PRs in the next couple of weeks.