[Feature Request][RFC] Multi-tenancy as a construct in OpenSearch

[msfroh]

Is your feature request related to a problem? Please describe

I've been involved with multiple projects and issues recently that try deal with the notion of "multi-tenancy" in OpenSearch. That is, they are concerned with identifying, categorizing, managing, and analyzing subsets of traffic hitting an OpenSearch cluster -- usually based on the some information about the source of the traffic (a particular application, a user, a class of users, a particular workload).

Examples include:

1. [PROPOSAL][Query Sandboxing] Query Sandboxing high level approach. #11173 -- Query sandboxing aims to prevent some subset(s) of traffic from overwhelming the cluster or individual nodes, by imposing resource limits.
2. [RFC] Query insights framework  #11429 -- Query insights can already capture the top N most expensive queries, but it will be more helpful once we can associate queries with a source. (This may also become an input to sandboxing decisions down the line.)
3. [Search Pipelines] Add a processor that provides fine-grained control over what queries are allowed #10938 -- Some OpenSearch administrators have asked for a feature that would let them restrict what kind of queries specific users/groups are allowed to send. In order to that, we first need to identify the users/groups.
4. [RFC] User Behavior Insights #12084 -- User behavior logging is a little different, because we expect it to deal with a large number of users who probably aren't hitting the cluster directly. This is aimed users of a search application, where the search application hits the cluster.
5. Slow logs -- While we don't have an open issue for it (I think), it would be worthwhile to include information about the source of a query in slow logs.

Across these different areas, we've proposed various slightly different ways of identifying the source of traffic to feed the proposed features.

I would like to propose that we first solve the problem of associating traffic with a specific user/workload (collectively "source").

Describe the solution you'd like

We've discussed various approaches to labeling the source of search traffic.

Let the client do it

In user behavior logging, the traffic is coming from a search application, which can presumably identify a user that has logged in to the search application. The application can provide identifying information in the body of a search request. This would also work for any other workload where the administrator has control over the clients that call OpenSearch.

Pros:

• Very easy to implement in OpenSearch. We just need to add a new property to SearchRequest (or more likely SearchSourceBuilder, since it probably belongs in the body). In the simplest case, this property could just be a string. For more flexibility (e.g. to support the full suite of attributes in the UBI proposal), the property could be an object sent as JSON. Of course, as these labeling properties grow more complex, it also becomes harder for downstream consumers (like query insights) to know which object fields are relevant for categorization.
• For application builders, it provides a lot of flexibility. Application builders know how they want to categorize workloads for later analysis.

Cons:

• Doesn't work in an environment where administrators have granted direct cluster access to many users, since they can't assume that users will provide accurate identifying labels.

Rule-based labeling

This is the approach that @kaushalmahi12 proposed in his query sandboxing RFC. A component running on the cluster will inspect the incoming search request and assign a label. (Okay, in that proposal, it would assign a sandbox, but it's the same idea targeted to that specific feature.)

Pros:

• Does not require changes to application code.
• Does not rely on trusting clients of the cluster to do the right thing.
• We could provide some sensible defaults. The coordinator node could tag the request with the source IP, for example. It would be great if we could take user identity information from the security plugin, but @peternied keeps telling me that's harder than it sounds (since the identity information might be a monstrous certificate).

Cons:

• Developing a rule engine (however simple) is more complicated than just adding a property to a search request.
• Need to worry about rule precedence.

Custom endpoints for different workloads

This is an evolution of what @peternied proposed in his RFC on views. Over on opensearch-project/security#4069, I linked a Google doc with my proposal for an entity that combines authorization (to the entity), access-control (to a set of indices or index patterns), document-level security (via a filter query), query restrictions, sandbox association, and more.

Pros:

• Easy to administer. For a given tenant, everything is defined in one place.
• No ambiguity. No conflicts. When you search via the given endpoint, the specified behavior is exactly what runs.
• Reduces authorization load for the security plugin. No need to check every single index/alias when you're defining access control on the endpoint.

Cons:

• Requires an endpoint per differentiated workload. Obviously won't work for something like UBI, where we're trying to learn from a larger user base. Gives a lot of power to administrators, but also a lot of responsibility. Assuming we store these endpoint definitions in cluster state (or any structure available to all coordinator nodes), you probably can't configure more than a few dozen or hundreds of them.
• Administrators may need to worry about index patterns "accidentally" picking up access to new indices.

What do I recommend?

All of it! Or at least all of it in a phased approach, where we learn as we go. The above proposals are not mutually-exclusive and I can easily imagine scenarios where each is the best option. In particular, if we deliver the "Let the client do it" solution, we immediately unblock all the downstream projects, since all of the proposed options essentially boil down to reacting to labels attached to the SearchRequest (or more likely SearchSourceBuilder).

I think we should start with the first one (Let the client do it), since it's easy to implement. The rule-based approach can coexist, since it runs server-side and can override any client-provided information (or fail the request if the client is trying to be sneaky). I would recommend that as a fast-follow.

The last option is (IMO) nice to have, but limited to a somewhat niche set of installations. It's probably overkill for a small cluster with a few different sources of traffic, but it would be helpful for enterprise use-cases, where it's important to know exactly how a given tenant workload will behave.

Related component

Search

Describe alternatives you've considered

The above discussion covers three alternatives and suggests doing all three. If anyone else has suggestions for other alternatives, please comment!

What about indexing?

I only covered searches above, but there may be some value in applying the same logic to indexing, to identify workloads that are putting undue load on the cluster by sending too many and/or excessively large documents. My preferred approach to avoiding load from indexing is flipping the model from push-based to pull-based (so indexers manage their own load), but that's probably not going to happen any time soon. Also, a pull-based approach means that excessive traffic leads to indexing delays instead of indexers collapsing under load -- you still want to find out who is causing the delays.

@Bukhtawar, you're our resident indexing expert. Do you think we might be able to apply any of the approaches above to indexing? Ideally whatever we define for search would have a clear mirror implementation on the indexing side to provide a consistent user experience.

[ansjcy]

Great proposal! From the search visibility point of view, currently the closest thing we can get about the "tenancy" of a search request relies on the thread context injected by the security plugin. The challenge here (also potentially the challenge to implement "default rules" mentioned in the RFC) is, the user has to use the Security plugin (Or we need to implement the methods to get "user" info for all identity plugins) and additionally, in VPC domains, the client IP obtained from the thread context is typically the ip of the Load Balancer (rather than the "real client ip"). We also briefly discussed some of those challenges in #12740 and #12529.

We really need a way to define multi-tenancy as the first class feature in OpenSearch, which allows us to define "tenancy" across various layers. We can start from multi-tenancy for search queries to solve issues mentioned in query sandboxing and query insights. And we should design the solution flexible enough so that it can be reused in indexing and plugins as well - The labeling mechanism outlined in the RFC can be a great first step for achieving full multi-tenancy in OpenSearch :)

Edit: Adding a very simple workflow diagram for client-side and rule-based labeling solutions, based on my understanding of this RFC:

Also, after chatting with .Jon-AtAWS about this topic, I wanted to add several things to keep in mind for tenant labeling:

• We should limit the number of labels to avoid excessive propagation throughout the search phase. Setting an upper limit for labels may be necessary.
• Ensure optimal performance for users opting not to use labels ("zero label"). Performance should not be compromised for users who choose not to leverage this feature.

[ansjcy]

Also @msfroh, how is

Custom endpoints for different workloads

really related to defining the tenancy info (or, associating traffic with a specific user/workload )? Please correct me if I'm wrong - I think it's more of a new way for managing a group of configuration to achieve "authorization, access-control, document-level security, query restrictions and more" for multi-tenant use cases?

[msfroh]

Also @msfroh, how is

Custom endpoints for different workloads

really related to defining the tenancy info (or, associating traffic with a specific user/workload )?

It's a way of explicitly defining the tenancy via the endpoint. If every tenant gets their own endpoint, it's server-side (so we don't need to trust the clients), but the resolution is easy/well-defined (versus applying rules).

We could just use the endpoints to apply tenant labels, but having the "all-in-one" configuration is a bonus, IMO.

[dblock]

Is there a variation of option 1 where the client is involved with establishing a session like typical web servers (requests without a session get a cookie from the server which is then passed back/around)?

[Bukhtawar]

Thanks @msfroh I like the proposal where it says the tags/identity etc aren't mutually exclusive. We can define ways to define an identity or a user-label or for that matter any other attribute associated with a request while keeping some of it auto-created like identity if the user is using a security plugin or a tab if the client is passing certain attributes.
For features like query sandboxing, we could define a sandbox rule and associate it either with a tag or identity to provide the right search experience to end customers while restricting/throttling limited set of bad identities

From an indexing perspective I would try to see if we could use something like #12683 or likes of data stream/index template to define the tenancy at an index level.
Apart from just indexing and search, we need to think about few isolation boundaries. Here I feel we also need to consider access permissions like separate index level encryption keys then decisions like pods/cell within a cluster as how data(indices for a given tenant) can be placed across these pods for logical isolation.

[msfroh]

Is there a variation of option 1 where the client is involved with establishing a session like typical web servers (requests without a session get a cookie from the server which is then passed back/around)?

That would be doable. I'm not immediately seeing how it helps for the multi-tenancy case -- as far as I can tell, it would help track "client that sent request A also sent requests B, C, D, and E". I'm not sure how the downstream components (query insights, slow logs, etc.) would use that information.

[peternied]

Custom endpoints for different workloads

@msfroh Thanks for the great write up. Couple of comments in third form of this construct:

• Scalability: This is an interesting consideration, but it seems inline with maximum limits on the number of index in a cluster. For a true enterprise 'multi-tenant' solution I think there will need to be considerable effort to decouple and rescale OpenSearch. I think this would be good to area to defer :D
• "Gives a lot of power to administrators, but also a lot of responsibility": food for thought: admins largely have this power today, but it isn't easy to comprehend. Being more transparent about what dials and knobs are available for the service is a big win IMO with little downside.
• Viability: Views was built as a direct mapping from the view name on to a fixed set of targets for a search request. It would not take much efforts to rename and modify the views systems to allow for a named set of targets to reshape the existing feature into _tentant/{my-tenant}/_target/{what-previously-was-the-view-name}/_search. I suspect that we'd want to do some additional research into what customers think about the trade-offs for additional endpoints vs configurability.

[kaushalmahi12]

Thanks for taking the time to write this up @msfroh!

Let the client do it

I am guessing the new object in the SearchRequest will still be a closed schema object (I mean there wouldn't be random fields coming in. e,g; it shouldn't be a Map<String, String>). Since this could potentially increase the memory footprint in the cluster for search heavy workloads.

Rule-based labeling

Regarding this I don't think this has to have affinity towards the sandboxing feature. Since rules will be an entity and can govern other actions as well like deciding whether to assign a label for features such as Sandboxing, QueryInsights or slow logs.