[8.0] [8.1] Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden

[mtojek]

Hi Team,

this weekend we managed to build a new Docker image for Kibana, which was supposed to include a few fixes we're waiting for. Unfortunately, it brought another problem:

Attaching to elastic-package-stack_kibana_1
�[36mkibana_1                     |�[0m [2022-01-16T23:40:05.855+00:00][INFO ][plugins-service] Plugin "metricsEntities" is disabled.
�[36mkibana_1                     |�[0m [2022-01-16T23:40:05.910+00:00][FATAL][root] Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html
�[36mkibana_1                     |�[0m     at ensureValidConfiguration (/usr/share/kibana/src/core/server/config/ensure_valid_configuration.js:25:11)
�[36mkibana_1                     |�[0m     at Server.preboot (/usr/share/kibana/src/core/server/server.js:160:5)
�[36mkibana_1                     |�[0m     at Root.preboot (/usr/share/kibana/src/core/server/root/index.js:48:14)
�[36mkibana_1                     |�[0m     at bootstrap (/usr/share/kibana/src/core/server/bootstrap.js:99:9)
�[36mkibana_1                     |�[0m     at Command.<anonymous> (/usr/share/kibana/src/cli/serve/serve.js:216:5)
�[36mkibana_1                     |�[0m 
�[36mkibana_1                     |�[0m  FATAL  Error: [config validation of [elasticsearch].username]: value of "elastic" is forbidden. This is a superuser account that cannot write to system indices that Kibana needs to function. Use a service account token instead. Learn more: https://www.elastic.co/guide/en/elasticsearch/reference/8.0/service-accounts.html
�[36mkibana_1                     |�[0m 

It's a huge blocker for Elastic Integrations, as we're unable to start the development stack 8.0-SNAPSHOT and 8.1-SNAPSHOT. Last time we has a similar discussion with the agent team to give us some slack to switch to service tokens.

Kindly please to revert this change due to its impact.

Impact:

https://beats-ci.elastic.co/blue/organizations/jenkins/Ingest-manager%2Fintegrations/detail/master/1430/pipeline (all integrations marked as broken)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[mtojek]

@jportner This PR caused a lot of troubles for us. Would you mind reverting it temporarily?

cc @pgayvallet @rudolf

[pgayvallet]

Would you mind reverting it temporarily?

How much time are we talking about here? This has to go with 8.0.0-GA and 8.1.0

[mtojek]

What's the safe period on your side? We need ~1 week to research it and switch to service tokens.

EDIT:

Hopefully it will be done sooner, but we need to deploy it in few places and might be concerned about stability. The elastic-package stack depends on Docker Compose. We may need to mount the $ES_HOME/config/service_tokens in the Elasticsearch image as we don't want to generate them every time (don't need to).

[pgayvallet]

One week would probably be fine, but I'll let @jportner have the final word on this one.

Btw, why can't you just switch to using the kibana user instead? This one only logs a warning and would work.

[mtojek]

Btw, why can't you just switch to using the kibana user instead? This one only logs a warning and would work.

Let me check that if it's fine.

[mtojek]

@pgayvallet I tried your suggestion to use kibana_system and it failed with:

�[36mkibana_1                     |�[0m [2022-01-17T08:29:14.627+00:00][ERROR][elasticsearch-service] Unable to retrieve version information from Elasticsearch nodes. security_exception: [security_exception] Reason: unable to authenticate user [kibana_system] for REST request [/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip]

Is it something we can easily workaround with config?

EDIT:

I see now, we need to use elastic user to set password for other users. In our case, it means that we need to intercept Docker Compose booting procedure with the command call, so it doesn't differ much from setting up a service token.

EDIT2:

I tried to use elasticsearch-service-tokens create elastic/kibana kibana-token-2, but it fails with:

elasticsearch_1              | {"type": "server", "timestamp": "2022-01-17T09:15:21,678Z", "level": "ERROR", "component": "o.e.x.s.a.s.FileServiceAccountTokenStore", "cluster.name": "elasticsearch", "node.name": "caf8265d5de5", "message": "failed to parse service tokens file [/usr/share/elasticsearch/config/service_tokens]. skipping/removing all tokens...", "cluster.uuid": "PkT_gzAWTai-ddYEck9cag", "node.id": "Re1wsc7XRfK9R1R7AnBrkQ"  }

[mtojek]

Ok, so I tried to switch to service tokens and failed with:

�[36mfleet-server_1               |�[0m Kibana Fleet setup failed: http POST request to http://kibana:5601/api/fleet/setup fails: Access to Fleet API requires the superuser role and for stack security features to be enabled.: <nil>. Response: {"statusCode":403,"error":"Forbidden","message":"Access to Fleet API requires the superuser role and for stack security features to be enabled."}

@pgayvallet Is there something I missed here?

[jportner]

@mtojek i made the change to fail fast at config time because Kibana will not work anymore due when configured with the elastic super user — this is due to an elasticsearch change (elastic/elasticsearch#81400).
I’m pretty sure if I reverted that PR, you’d still have a problem on your hands 😔

[mtojek]

Just an update on our side -

We managed to implement the logic around service tokens and it appears to be working. I suppose that you don't need to revert any commits :) Thanks for your cooperation.

I will resolve the issue.

[yitzchake]

@mtojek , can you list the command you used to implement the service token? I'm running into the same isssue