[Windows] Configuration values are improperly decoded

[peternied]

What is the bug?
When users.yml is proccessed by DynamicSecurityConfig through the scenario tests the user strings are improperly treated as ANSI values.

Expected username §ÄÖÜäöüß, malformed username §ÄÖÜäöüß

How can one reproduce the bug?
On a windows machine run the following command
./gradlew.bat test --tests org.opensearch.security.IntegrationTests.testSpecialUsernames

Additional Context
Its not yet clear if this is a scenario test issue or if this a problem with any configuration that is loaded normally.

Should attempt a manual repro on windows with an UTF-8 name like above in the configuration settings.

[stephen-crawford]

So I have some news on this issue:

I took several steps to try to determine the most likely cause and believe I have narrowed it down.

Right now, the integration test for the special usernames functions perfectly fine on-non special usernames for any internal users in the internal_users yaml file. This suggests that the issue is not a universal encoding error but instead an issue arising from the conversions of Windows-1252 into UTF-8 and vice versa.

What complicates things is the location of this translation. When running Opensearch 2.4 with the security plugin, you can access the node information ONLY when you log in as one of the original internal user roles: admin, readall, kibanauser, etc. So while the security plugin successfully tests on users such as rexclude the Opensearch node responds 401 from a manual attempt -- likewise for any of the special names.

Going deeper, it appears somewhere OpenSearch is populating from only the internal_users2.yml file instead of both. I am not sure where yet but I do know that this is likely related to at least one side of the problem. The next step will be to check whether moving a role from internal_users.yml to 2.yml makes it identifiable by OpenSearch on login and then further if this somehow helps with the special character issue.

[peternied]

@scrawfor99 Setting aside the integrations tests cases, can you manually reproduce the issue with a internal_users.yml bootstrapped cluster?

Separately did you see that in base window OpenSearch (no security plugin), if you modify the config's cluster.name value to §ÄÖÜäöüß, when you call the information endpoint at / it comes back without being malformed, like below? [1]

GET http://localhost:9200/
{
  "name" : "opensearch-node1-2-2-0",
  "cluster_name" : "§ÄÖÜäöüß",
  "cluster_uuid" : "9jZU_R4_RH-98h15iCmhsA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "2.2.0",
    "build_type" : "tar",
    "build_hash" : "b1017fa3b9a1c781d4f34ecee411e0cdf930a515",
    "build_date" : "2022-08-09T02:27:25.256769336Z",
    "build_snapshot" : false,
    "lucene_version" : "9.3.0",
    "minimum_wire_compatibility_version" : "7.10.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

[1] https://opensearch.org/docs/latest/opensearch/configuration/

[stephen-crawford]

In response to the second question, the cluster.name does come back correctly formatted when you query the address even when there are special characters.