[Extensions] Add `extensions/extensions.yml` setting to enable backward compatible plugin mode for extensions

[cwperks]

A new setting in opensearch.yml extensions/extensions.yml needs to be added to enable backward compatible plugin mode for extensions. The setting can be:

extensions:
  - name: hello-world
    uniqueId: hw
    hostAddress: '127.0.0.1'
    port: '4532'
    version: '1.0'
    opensearchVersion: '3.0.0'
    minimumCompatibleVersion: '3.0.0'
    bwcPluginMode: true

Background

Many plugins currently rely on user information to be present on the ThreadContext to support plugin use-cases. Examples include:

1. resource protection - i.e. sharing detectors in Anomaly Detection via backend roles)
2. asynchronous operation -i.e. running a detector on a schedule and injecting the user's roles from the time of detector creation to evaluate if a user has privileges on an index

See: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/privileges/PrivilegesEvaluator.java#L198-L210

private void setUserInfoInThreadContext(User user, Set<String> mappedRoles) {
    if (threadContext.getTransient(OPENDISTRO_SECURITY_USER_INFO_THREAD_CONTEXT) == null) {
        StringJoiner joiner = new StringJoiner("|");
        joiner.add(user.getName());
        joiner.add(String.join(",", user.getRoles()));
        joiner.add(String.join(",", Sets.union(user.getSecurityRoles(), mappedRoles)));
        String requestedTenant = user.getRequestedTenant();
        if (!Strings.isNullOrEmpty(requestedTenant)) {
            joiner.add(requestedTenant);
        }
        threadContext.putTransient(OPENDISTRO_SECURITY_USER_INFO_THREAD_CONTEXT, joiner.toString());
    }
}

In order for extensions to initially be a replacement for plugins and support all current use-cases, they will need the data from the above snippet. (Note: specifically the information needed is the username, backend_roles and mapped_roles - mapped roles here refers to the ultimate roles that have been resolved to after all roles mapping - 1. roles explicitly assigned + 2. roles mapped to via backend_roles 3. roles matching the hosts in roles_mapping and 4. And_backend_roles - the user needs all backend roles in the list to be mapped. Full Details: https://github.com/opensearch-project/security/blob/main/src/main/java/org/opensearch/security/securityconf/ConfigModelV7.java#L1206-L1269)

This setting will ultimately determine if the encryption_key from #2615 is used to encrypt the roles and backend_roles claims in the payload of the token sent to an extension.

See design considerations

• [Question] How to support extensions parity for user information #2536

[cwperks]

@peternied Made a good point that this should be configured on the extension level and not globally for all extensions. A good place for this to be is in extensions/extensions.yml. See example in the SDK Dev guide: https://github.com/opensearch-project/opensearch-sdk-java/blob/main/DEVELOPER_GUIDE.md#create-extensionsyml-file

[stephen-crawford]

[Triage] This is part of the Extensions project.