[FEATURE] New reserved roles are not sourced to security index on an upgrade

[cwperks]

Is your feature request related to a problem?

New reserved roles are often added between releases when a new plug-in is released and associated roles are created to provision access to features of the new plugin. Reserved roles are added to the security repo roles.yml like is done in this PR: #2192 which introduces 3 new reserved roles between 2.3.0 and 2.4.0.

When starting up a cluster with the upgraded version of OpenSearch I receive the following line:

Index .opendistro_security already contains doc with id roles, skipping update

indicating that the new reserved roles in roles.yml will need to be manually added either via the API by an admin or via securityadmin tool connecting via clientcert as a super admin.

This becomes additionally complicated when users of OpenSearch add custom roles or modify the security index in any way via the OSD UI or API as it is destructive to run securityadmin.sh because it can delete custom configuration. In order to get around this I have:

1. Ran a backup of the security index using security admin: https://opensearch.org/docs/latest/security-plugin/configuration/security-admin/#a-word-of-caution
2. Reconcile the exported roles.yml with new reserved roles in the repo's roles.yml: https://github.com/opensearch-project/security/blob/main/config/roles.yml
3. Run securityadmin.sh with the coalesced exported roles.yml with custom roles and the repo's roles.yml of the corresponding version of opensearch

This is not an idea process and there can be a better solution to source new reserved roles during an upgrade

[peternied]

@opensearch-project/security-analytics This scenario might be impacting y'all, it might be worthwhile to invest in this scenario. Here is where I believe there is a lot of overlap, after upgrading a cluster that does not have the new roles needed for the security-analytics plugin when it is 'upgraded-in' on at a later time.

[cwperks]

[Triage] In the short-term this can be a documentation fix.

@peternied to follow-up on this issue.

[eirsep]

One possible solution:
Have 2 security config indices - one for pre-defined roles (internal to Opensearch and its plugins) and one for execlusively for user defined roles
On every node restart/bootup (or a new admin API invocation), we should reload roles/permissions into the pre-defined roles index from the yml file and leave the user security roles as is.

This would remove some of the static-ness from our design and solve the problem for upgrading domains getting the latest roles and permissions from yml file without user action/documentation.

[peterzhuamazon]

This is observed on a cluster where the cluster was upgrade from 1.3 to 2.3 and missing CCR perms and roles. This does not resolve after redeploy.

[peternied]

This is observed on a cluster where the cluster was upgrade from 1.3 to 2.3 and missing CCR perms and roles. This does not resolve after redeploy.

This is the expected behavior, the security plugin does not alter the security configuration on established clusters. Cluster administrators can add this functionality via the security admin tool https://opensearch.org/docs/latest/security/configuration/security-admin

This feature request is about adding this functionality which some administrators might be comfortable with, whereas others might not want this behavior. Its a complex scenario based on the expectations of the administrator, we would be open for pull requests around this space.

[peterzhuamazon]

This is observed on a cluster where the cluster was upgrade from 1.3 to 2.3 and missing CCR perms and roles. This does not resolve after redeploy.

This is the expected behavior, the security plugin does not alter the security configuration on established clusters. Cluster administrators can add this functionality via the security admin tool https://opensearch.org/docs/latest/security/configuration/security-admin

This feature request is about adding this functionality which some administrators might be comfortable with, whereas others might not want this behavior. Its a complex scenario based on the expectations of the administrator, we would be open for pull requests around this space.

Got it, it would be nice if it can prompt a windows to let me know new roles in 2.3 version after upgrade. This would be very handy and easy for the users.

Without this, I was fairly confused for half a day trying to figure out is my cluster have enough permission to run ccr, with multiple b/g and upgrade to latest version, without using the all_access role.

Thanks.

[peternied]

@peterzhuamazon Roger that - must have been frustrating. Good idea about the warning.

[peternied]

Proposal - Check/Upgrade reserved roles from config

OpenSearch's security defaults are controlled by the ./config/.* files, including roles. This file contains many reserved roles that are updated by plugins for in OpenSearch distribution. By checking the reserved roles against the roles of the cluster we could detect that an update to the reserved roles might be needed and also apply an update resync these roles.

Updated eligible check on startup

On the security plugin start - for only the master node, read the config/roles.yml file into memory, compare that all roles on disk exist, if not then suggest an upgrade. Then check that all roles have the same permissions, if they do not then suggest an upgrade. Do this as a single warning message that includes the REST API to use to perform this upgrade - [WARN] Reserved roles do not match the defaults from ./config/roles.yml. Resync these roles by calling "POST /_plugins/_security/api/_reserved/_upgrade_apply".

Upgrade eligible check via API

Create an API "GET /_plugins/_security/api/_reserved/_upgrade_check" that can be used by tools to determine the upgrade state with the following responses.

200 OK
{
   "canUpgrade": true
   "roles": {
       "missingRoles": [
            // Full role definition
           "{ROLE-NAME}: {
              "cluster_permissions": ...
           }
       ],
       "updatedRoles": [
            ...
       ]
    }
}

Upgrade action via API

Create an API "POST /_plugins/_security/api/_reserved/_upgrade_apply" that triggers an upgrade, overrides all permissions on reserved roles and replaces them with what is found in the config/roles.yml file. Internally this will be handled the same as a PATCH request to the roles API

200 OK
{
  "status": "OK",
  "message": "Reserved roles '<role>,...' added, '<role>,...' updated."
}

Security admin tool

Add additional options to the security admin tool to include options to check this API and perform the upgrade.

[cwperks]

@peternied This approach looks good to me. I think it makes sense to create an API where you can check to see if your current security config has the latest version of all reserved rolls.

I know there is no current mapping between role -> plugin, but if its possible, it would also be interesting to take into account whether a specific plugin is installed and that the roles associated with the plugins exist in the security index.