Point in time security model

[bharath-techie]

Point in time security model

Context:

What is point in time ?

User can create a Point In Time for a set of indices. A Point In Time is a set of segments which are frozen in time taken at the moment of creation. We lock the segments of those indices’ shards and create contexts to access and query only those shards.

Such point in time contexts can be reused across multiple search queries and the data that is searched upon will be on the locked segments. This solution is used with search after for pagination with data consistency.

This can be related to ‘scroll’ , whereas ‘scroll’ context gets created for each query ( cannot be reused for different search queries ) and ‘scroll’ can only move forward.
Design doc : opensearch-project/OpenSearch#3960
Meta issue: opensearch-project/OpenSearch#3959

Point in time APIs

• Create point in time
  • The request implements ‘IndicesRequest.Replaceable“
  • action - “indices:data/read/point_in_time/create”
  • User can create point in time on any index for which the user has access to ( similar to search )
• Delete point in time
  • User can pass list of PITs to delete (or) user can choose to delete all PITs
  • action - “indices:data/read/point_in_time/delete”
• List all PITs
  • This API lists all PITs that the user has access to
  • Rest user facing action to list PITs- “indices:data/read/point_in_time/readall”
  • Action to retreive all cluster PITs - “cluster:admin/point_in_time/readall”
• Point in time segments API
  • This shows the resource utilization of segments held because of a point in time context.
  • User can pass list of PITs to show segments info (or) user can choose to show all PIT segments info
  • Action - "indices:monitor/segments/point_in_time"

Permission model

Action group:

manage_point_in_time:
  reserved: true
  hidden: false
  static: true
  allowed_actions:
    - "indices:data/read/point_in_time/create"
    - "indices:data/read/point_in_time/delete"
    - "indices:data/read/point_in_time/readall"
    - "cluster:admin/point_in_time/readall"
    - "indices:monitor/point_in_time/segments"
  type: "index"
  description: "Manage point in time actions"

Role which has access to all PIT actions :

point_in_time_1:
  reserved: true
  hidden: false
  description: "Migrated from v6 (all types mapped)"
  cluster_permissions:
    - "cluster:admin/point_in_time/read_from_nodes"
  index_permissions:
    - index_patterns:
        - "pit_1"
      dls: null
      fls: null
      masked_fields: null
      allowed_actions:
        - "manage_point_in_time"

Requirement

• By design, the user is able to ‘create pit’ on any index and ‘search’ using the same PIT if the underlying index permissions are present
• So the user should be able to delete or list the PITs which they were able to create ( as long as the underlying index permission is present )
• User should not be restricted to these APIs (delete, list or segments ) if they don’t have all indices read permission

In order to meet the above requirements, we are making changes in security plugin to evaluate permissions of PITs.

Following two items need review:

1. Proposed security model
2. List all PITs design

1. Proposed security model

We need to handle two cases for the APIs

• Access for explicit PITs ( LIST request )
• Access for ‘all’ type PIT operations
  • Access when no PITs are present in cluster

When is a PIT permitted ?

A PIT is made of one ore many indices. If the user has access to all the indices of a PIT, then PIT is permitted.
This is common for all APIs.

What if user does not have action permissions ?

Operation is denied.

a. Access for explicit PITs

• All PITs in the request must be permitted, then we allow the operation ( ALLOW )
• Even if one PIT is not permitted, we deny the operation ( DENY )
• This is similar to search and other indices APIs ( user must have access to all explicit indices given as part of search request )

b. Access for operation with ‘all’ or ‘ * ’ PITs

• Out of all PITs in cluster , if any PIT is permitted , then operation will carry forward ( ALLOW )
• If there are no permitted PITs, throw forbidden error ( DENY )
• This is inline with search API

No PITs in cluster

• General case, Deny operation. Throw 403 - forbidden error ( DENY )
• Special case - Similar to other indices operation, if the config property - do_not_fail_on_forbidden_empty is true - then allow operation ( ALLOW )

@Override
    public boolean isDnfofForEmptyResultsEnabled() {
        return config.dynamic.do_not_fail_on_forbidden_empty;
    }

2. List all PITs Design

Context

• We need to get all the PITs in the cluster before filtering them.
• Similar operation is node stats or node hot threads. ( as we broadcast to all the nodes to get data )
• For further context please refer to this thread - Added rest layer changes for List all PITs and PIT segments OpenSearch#4388 (comment)

There are two options currently at discussion

Option 1 : New action filter for PITs in security plugin

We will introduce a new action filter which will be called by security filter for PIT requests to wrap the listener to call the PIT privilege evaluator to get the permitted PITs.
Check this thread : opensearch-project/OpenSearch#4388 (comment)

Pros :
No extra action / permission required

Cons :
Maintainability of additional filter code in security plugin

Option 2 : Dummy action to filter PITs

Dummy action in opensearch to filter PITs

Pros :
Simple solution, doesn't add any code to security plugin and it is already tested in the current PR

Cons :

• Extra dummy action is used for nothing other than filtering when security plugin is enabled. No business logic is present.
• Extra action permission required for user

[bharath-techie]

@peternied @cliu123 please review this.

@reta @Bukhtawar

[cwperks]

Assigning this issue to @peternied to provide next steps.

[peternied]

@bharath-techie Thanks for writing this up and its really useful to have available. This is a really meaty document with many questions and assumption inside it - this is making it hard for me to build up a model for how this system works. It will be faster to build on smaller scenarios first as we move towards encapsulating all scenarios you'd like to support.

I am having serious issues with the REST API layout, as it isn't following a convention of an addressable resource.

Creation is done via POST /index-1,index-2,index-N/_point_in_time
There is no single retrieval for the PIT instance.
DELETE /_search/point_in_time is built off of search action.

This is creating problems describing the security posture because our typically conventions are grounded in addressable objects. Can we make the access patterns look like the following, and then create straight forward permissions off of them?

POST /_point_in_time
{
   "indices": ["index-1", "index-2", "index-N"],
   "creationTime": 1658146050064 
}

{
   "id": "FOO_BAR",
   "indices": ["index-1", "index-2", "index-N"],
   "creationTime": 1658146050064,
    "_shards": {
        "total": 3,
        "successful": 3,
        "skipped": 0,
        "failed": 0
    }
}

GET /_point_in_time/{id}

{
   "id": "FOO_BAR",
   "indices": ["index-1", "index-2", "index-N"],
   "creationTime": 1658146050064 
}

DELETE /_point_in_time/{id}

GET /_point_in_time/_search?q=index-1
{
  "hits": {...}
}

[peternied]

From List all PITs Design

We will introduce a new action filter which will be called by security filter for PIT requests to wrap the listener to call the PIT privilege evaluator to get the permitted PITs.

I'm not sure I understand the argument against writing the additional code. Using a PIT focus action filter seems like a good separation of concerns.

[bharath-techie]

create

List all PITs API is exposed so that user can view all PITs assigned to them. (cat indices for example ) So the need of the API is to get the IDs and additional info - so we can't pass ID to the api.

For delete,
We are not doing something different from existing APIs.
https://opensearch.org/docs/latest/opensearch/rest-api/scroll/ -- scroll supports deleting all and deleting list ( ids are passed in body or in url )

Also these APIs are designed keeping opensearch dashboards use cases as well in mind, so we need all these patterns to be supported.

[peternied]

Lets focus in on where PIT is closely related to scroll and set aside the cluster/admin* scenarios:

Scroll has fewer permissions indices:data/read/scroll and indices:data/read/scroll/clear. Why do we need the additional permissions?

Looking at the capabilities of a PIT object and what warrants protection:

• Creation PIT objects allows for allocation of resources on the cluster, safe guarding this with a permission indices:data/read/point_in_time/create looks well justified
• Deletion of a PIT can cause request to fail that depend on it, so the lifecycle should be guarded as well, indices:data/read/point_in_time/delete looks well justified
• Enumeration, having the id of PIT discloses the indices its associated with - I believe this is why there is the statement If the user has access to all the indices of a PIT, then PIT is permitted. - could we alter the PIT id not to include this information? Then there is no need to protect it.

If the PIT does not include the index name inside of it, as the use case of PIT is tightly coupled with _search action on an index that permissions check will implicitly occur, and if the PIT is not associated with it the request will fail.

If this understanding of the scenario isn't accurate, please provide scenarios for why an admin would want additional permissions granularity.

[bharath-techie]

So you mean to say list all PITs should be a cluster admin operation and no need of granularity to it ?

Only concern is that let's say 'user a' has access to 'index a' and created a pit for it.

If 'user b' has 'list all pits' permissions ( cluster admin permission) but don't have access to any indices, still 'user b' will be able to read the PITs info 'user a' created.

Is that acceptable behavior ?

[bharath-techie]

GET /_search
{
  "size": 10000,
  "query": {
    "match" : {
      "user.id" : "elkbee"
    }
  },
  "pit": {
    "id":  "46ToAwMDaWR5BXV1aWQyKwZub2RlXzMAAAAAAAAAACoBYwADaWR4BXV1aWQxAgZub2RlXzEAAAAAAAAAAAEBYQADaWR5BXV1aWQyKgZub2RlXzIAAAAAAAAAAAwBYgACBXV1aWQyAAAFdXVpZDEAAQltYXRjaF9hbGw_gAAAAA==", 
    "keep_alive": "100m"
  },
  "sort": [ 
    {"@timestamp": {"order": "asc", "format": "strict_date_optional_time_nanos"}},
    {"_shard_doc": "desc"}
  ]
}

This is how search operation uses PIT as well, indices are not present as part of search operation, whenever PIT ID is used as PIT comprises of indices information.

So PIT will include index information as that's per the current design.

[reta]

Thanks @bharath-techie , I think with Option 1: New action filter for PITs in security plugin we could also eliminate the need to have "cluster:admin/point_in_time/readall" and only use "indices:data/read/point_in_time/readall", correct?

[bharath-techie]

No @reta if we need to use "indices:data/read/point_in_time/readall", user needs "all indices read permission", which we are trying to avoid.

Because "indices:data/read" actions either needs indices as part of request or user needs all indices read permission.

Only thing action filter will do is filter the permitted PITs which we will get after "cluster:admin/readpits" fetches all the pits.

So we can't get rid of "cluster:admin" call if we're going for granular model.