Allow Deprecation Info API to check for deprecated security configuration

[bizybot]

We are in the process of deprecating few Index privilege names that are currently in use. These may or may not have an alternative that is an exact match and so it could be breaking change when few of the deprecated privilege names are removed in future versions.

The index privilege names are referred to when creating a role for the user. The roles are indexed in the .security index and sometimes defined in the file.

The current deprecation info API captures information for cluster_settings, node_settings, index_settings, ml_settings. This seems to be more tied to Settings deprecation and maybe we could enhance this API to include additional deprecation information other than settings like the use of deprecated index privilege names in the roles from .security index or role file.

This issue exists to find if the current deprecation info API can be used for this purpose or if there are any other suggestions to handle this.

Relates: #47333

[elasticmachine]

Pinging @elastic/es-security (:Security/Authorization)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Features)

[jasontedor]

@bizybot I am making an assumption here that when these role names are used, we would emit a deprecation warning and log it. In which case, see #46106 and #46107. That’s going to be our general mechanism for exposing deprecated functionality without having to work too hard to acquire this information otherwise. Can you clarify if my assumption is correct and if what I propose in #46106 and #46107 would be acceptable as general infrastructure without having to build specific infrastructure for the situation here?

[bizybot]

Hi @jasontedor, Thank you for your comment. Indeed what you propose will help to showcase the deprecated usage of privilege names and that would be helpful for the user.
But the deprecation logging would happen only when the privilege names are referred mostly during authorization, so if the user does not log in and perform any action, the deprecation logs would be missing for this usage.

I assume that the Upgrade assistant acts on the info presented by deprecation info API (I might have the wrong assumption here), for example, if we scan .security index and find the roles that need to be migrated as they are using deprecated privilege names and then Upgrade assistant would look at that and invoke some migrate endpoints to perform migrations.
Would this be possible with the suggestions from #46106? I see that we will be collecting the deprecation logs, but unsure how it will be used by the upgrade assistant.

[jasontedor]

But the deprecation logging would happen only when the privilege names are referred mostly during authorization, so if the user does not log in and perform any action, the deprecation logs would be missing for this usage.

It's not clear to me why we need to emit a deprecation warning for each user, but rather when we parse the role descriptors. Can you explain?

I assume that the Upgrade assistant acts on the info presented by deprecation info API (I might have the wrong assumption here), for example, if we scan .security index and find the roles that need to be migrated as they are using deprecated privilege names and then Upgrade assistant would look at that and invoke some migrate endpoints to perform migrations.

The upgrade assistant offers suggestions that the user can employ to migrate from deprecated functionality. If it's possible to automatically upgrade instances in the .security index, we should do that internal to Elasticsearch automatically on behalf of the user, rather than expecting that they use the upgrade assistant to do so.

[gwbrown]

To clarify the technical aspects of this a bit: Each type of deprecation check (Node, Cluster, Index, ML) has a framework that takes each check as a function object, and passes it the appropriate type of metadata (IndexMetaData, ClusterState, etc.). Because role information is not stored in the cluster state, but rather can be in a combination of the .security index and file-based roles, we would need a new piece of infrastructure to retrieve the appropriate information from Security to allow checks to be written against it.

If we can automatically upgrade roles, we should do that, and ignore the rest of this post.

If we can't handle the necessary changes automatically, I think putting these checks in the Deprecation Info API has a big advantage: it allows users to interactively query the API to make sure they've fixed everything that needs fixing, as opposed to having some uncertainty about that - I mean, I'm not sure what would trigger deprecation logs for deprecated role names, and I'm an ES developer. Is it on node startup? Does it require a full cluster restart? If I was administrating a cluster I would feel very uncertain looking for a lack of deprecation logs to make sure I'd fixed the issue, which would be very frustrating given how much a broken security configuration can ruin your day.

To be clear: I'm absolutely for integrating the deprecation logs into the upgrade assistant. But I think for this case, if it's not too difficult, we should make this possible to check in the deprecation info API directly (assuming we can't just auto-fix it).

[jasontedor]

@gwbrown I think that there is potential for the roles defined in .security to be automatically upgraded (we need someone from the security team to confirm), but not those that are on disk in the roles file (because we aren't going to mutate those).

[bizybot]

@jasontedor @gwbrown

It's not clear to me why we need to emit a deprecation warning for each user, but rather when we parse the role descriptors. Can you explain?

Sorry I did not elaborate on my comment. Yes, you are correct that the deprecation logging would happen when we parse the role descriptors. The role descriptors are loaded when the user authenticates and the authorization engine loads the role descriptors for the role name(s).
So there is a possibility that the deprecation warning might be missing for some role names if the user with the role never logs in.

If we can automatically upgrade roles

I think we would not be able to automatically upgrade all the role descriptors, the reason being the source for the role descriptors can either be a file, .security index or custom roles provider.
Also, the deprecated role names might not have an exact replacement to take an automatic decision for migration and would need the user's inputs. +1 on the avoiding mutation of the file.

The integration of deprecation logs into the upgrade assistant will definitely help to showcase the warnings though there is an advantage in the deprecation info API as it can always parse the deprecations at one go from the .security and file roles store that allows users to see if there are any warnings related to it, instead of waiting for it be populated in the deprecation logs.

Note that both the ways are not going to emit all the deprecated usage there will be some misses for example if there is a custom roles provider deprecation info check function would not be able to parse the role descriptors generated by it or deprecation logging missing sometimes if the user does not log in.

[tvernum]

rather when we parse the role descriptors

I think the missing context is that we only load & parse index-based role descriptors when they are referenced in some way. That could be that a user with the role authenticates, or someone calls GET /_security/roles, but it is entirely normal for one or more roles to go for some period of time without being loaded/parsed.
We could forcibly load all role descriptors on node startup (or master change, or ...) but we don't currently do that.

I think that there is potential for the roles defined in .security to be automatically upgraded

I think we can, but I'm not sure we should, or at least I can't decide when we should do that.
My reasoning is:

• I think we should deprecate the old names as soon as we define their replacements so that it's clear why there are two sets of names and which ones should be used. That is likely to be in a minor release.
• For completeness, I would expect us to expose that deprecation in a number of places including the Roles UI, the Roles API, and deprecation logs
• We could automatically upgrade the security index in that release but it has some implications that are potentially confusing:
  1. It would mean that installing a minor upgrade would cause sudden, possibly unexpected, changes to their roles. This is likely to confuse users who are familiar with their existing privilege names and suddenly see that things are different.
  2. We would need to perform this same translation whenever the PUT /_security/role/{name} API was called so that the new/updated role didn't put the index back into a deprecated state. This would mean that PUT + GET would return a different JSON, which might cause problems for tooling.
• My gut feel is that users would prefer that we didn't apply automatic conversions of security data without them opting-in (especially in a minor).
• It would be reasonable to automatically upgrade the roles during a major upgrade when we remove the old names.
• However, giving users deprecation warnings in a minor, but not giving them the tools to fix them automatically until a major seem wrong. Some users will manually fix the roles when they see the warnings, only to find that there was a tool for that later in the release cycle.

Given the above, my thoughts were that the tool would be available, opt-in during the minor and then automatic at the major.

not those that are on disk in the roles file (because we aren't going to mutate those).

Correct.

custom roles provider

I don't know how much we can do for custom role providers. The custom roles extension API doesn't provide a way to list role only to get them by name, so it's impossible to know if an extension defines a role that uses a deprecated privilege name unless that role is referenced somewhere. The Upgrade Assistant won't be much help here since it won't be able to list custom roles that use deprecated privilege names.
My hope is that anyone who has a custom roles provider will test it when the recompile it for a major release, but I'm probably optimistic there.

deprecated role names might not have an exact replacement to take an automatic decision for migration

That's true in theory, but I don't think it is practically true in this case (per #47333 (comment), I think every existing privilege can be replacted with one-or-more of the new privileges).