[RFC] Search Query Sandboxing: User Experience

[kiranprakash154]

Co-Author @kkhatua

#8879 - [RFC] High Level Vision for Core Search in OpenSearch
#11061 - [RFC] Query Sandboxing for search requests

A common challenge with managing resources on OpenSearch clusters has been keeping runaway queries in check. With Search Backpressure, the ability to avoid running out of resources is now available, but there is no capability of protecting tenants who might be unfairly penalized. The goal of this RFC is to propose a mechanism for how admin users will be able to organize tenants into different groups (aka Sandboxes), and limit the cumulative resource utilization of these groups. We will mention an idea of how the sandbox is enforced, but will likely be a separate RFC due to its complexity.

What is a Sandbox ?

A sandbox is a logic construct designed for running search requests within the virtual limits. The sandboxing service will track and aggregate a node's resource usage for different sandboxes, and based on the mode of containment, limit or terminate tasks of a sandbox that's exceeding its limits.
A sandbox's definition is stored in cluster state, hence the limits that need to be enforced are available to nodes across the cluster.

Tenets

1. A sandbox can provide constraints on one or more resources.
2. A user may hint preference for a sandbox
3. A request can map to only one sandbox
4. A request that maps to multiple sandboxes will be assigned to a sandbox with the highest similarity score
5. A request maps to no sandbox or multiple sandboxes with the same similarity score will be assigned to a system-defined catchAll sandbox called genPop (general population).
6. The mapping attributes of a sandbox must be unique as compared to other sandboxes
7. The sandbox thresholds for a resource cumulatively cannot exceed 100%

Schemas

Sandbox Definition

The following is an abstract example of a sandbox’s definition, and is broadly broken into 4 essential elements within the document

{
    "name": "<nameOfSandbox>",
    "attributes": {
        "attributeA": "<condition1,condition2>",
        ...
    },
    "resources": {
        "resourceX": {
            "allocation": "<floatValue:(0-1)>", 
            "threshold": "<floatValue:(0-1)>" 
        },
        ...
    },    
    "enforcement": "<monitor|soft|hard>"
}

• name : defines a unique String identifier. Internally a UUID is used
• attributes : a collection of unique attributes that need to be matched by a request to be governed by that sandbox. We are considering user_name, indices_name, role, and custom_id
• resources : The set of unique resources for which this sandbox is being defined. We are considering jvm and cpu
• enforcement : How is the sandbox being enforced. This can be either monitor or soft or hard

Resource Definition

For each resource, a cluster level schema is also required, and the following is an abstract example

{
    "resources": {
        "resourceX": {
            "threshold": {
                "rejection" : "<floatValue:(0-1)>",
                "cancellation" : "<floatValue:(0-1)>"
            },
            "enforcement": "<monitor|soft|hard>"
        },
        ...
    }
}

• resources : The limits of a resource within which all the sandboxes must cumulatively operate before requests get rejected or cancelled. rejectionThreshold <= cancellationThreshold
• enforcement : How the containment of sandboxes for this resource will be managed. This can be either monitor or soft or hard , and can potentially override a sandbox’s enforcement.

High Level Flow

A request landing on a coordinator node will first need to be mapped to a sandbox, as per the tenets. Once a sandbox has been mapped, all child tasks spawned from the request will also inherit the sandbox allocation, irrespective on which node it runs.

Sandbox Resolution

The sandbox resolution happens on the coordinator node and will persist with all the tasks (and child tasks) of that request for the entirety of its lifecycle.

Thresholds Enforcement

As the sandboxes are enforced at a node level, for each resource, there are 2 thresholds defined cluster-wide:

• rejection — Reject new tasks if this threshold is breached
• cancellation — Cancel inflight tasks if this threshold is breached

Each Sandbox level thresholds are always proportional to the node level thresholds

The following is an example where

• 2 sandboxes have 20% and 25% thresholds for a resource, and the rest (100% - 20% - 25% = 55% ) is part of the general population (aka genPop) sandbox.
• If a node exceeds 85% usage for a resource, it will actively reject new tasks destined for the overflowing sandboxes.
• If the resource usage exceeds 95%, the node will actively cancel inflight tasks on the overflowing sandboxes

Additional Context

This RFC aims to discuss ideas at a high level. More details are provided in this google doc. Anyone with the link has comment access and we would love to gather feedback.

[kiranprakash154]

@msfroh @dblock @reta @sohami @Bukhtawar @nknize would love your thoughts !

[msfroh]

Could we piggy-back on the idea of "views" to enforce sandboxes? A view could be associated with a sandbox.

Alternatively, if we wanted to be more flexible, a view could enforce a search pipeline, then the search pipeline could (conditionally) modify the request to associate it with a sandbox.

[peternied]

[Triage - attendees 1 2 3 4 5]
@kiranprakash154 Thanks for creating this issue

[msfroh]

My next few messages contain notes from Search Backlog and Triage meeting: https://www.meetup.com/opensearch/events/298411954/. The @ references refer to the people who said the respective lines.

• @kkhatua: This is a feature aimed at sysadmins. We want to allow them to segregate their user classes and ration resources per class.
• @kiranprakash154: Enforcement has three levels: Monitor, soft, and hard enforcement. Monitor just gives you stats. Soft lets queries in until the node is under duress. Hard limit starts canceling queries when the sandbox is exceeded, whether the node is in duress or not.

Example from the Google doc that helps clarify things more than the example above:

Two Sandboxes defined for memory and CPU

[
 { 
   "name": "analysts",
   "attributes": {
       "role": "ba-*"
   },
   "resources": {
       "jvm": {
           "allocation": "0.5"
       },
       "cpu": {
           "allocation": "0.3"
       },
   },   
   "enforcement": "soft"
},
{
   "name": "operations",
   "attributes": {
       "role": "tellers",
       "indices_name": "transactions-*"
   },
   "resources": {
       "jvm": {
           "allocation": "0.25"
       }
   },   
   "enforcement": "hard"
 }
]

[reta]

So I think the usage of the sandbox concept is confusing here: we are not sandboxing anything (we cannot limit resources) but track the resource usage and react on that. Better names would be "resource limits group", "query prioritization group" or something along these lines.