APM Server not being able to authenticate with Elastic Server

[npapagna]

Hello,
I stumbled upon this issue while trying to use docker-compose to setup a 6.5.1 stack (modified the .env file for this).

Problem: I don't see activity in APM (Kibana) although the Java agent connects successfuly to the APM Server and is able to send data to it.

Diagnostic: running docker-compose logs apm-server shows Unauthorized (...) failed to authenticate user [elastic]:

apm_server       | 2018-11-29T20:06:57.694Z	INFO	pipeline/output.go:93	Attempting to reconnect to backoff(elasticsearch(https://elasticsearch:9200)) with 6 reconnect attempt(s)
apm_server       | 2018-11-29T20:08:01.988Z	ERROR	pipeline/output.go:100	Failed to connect to backoff(elasticsearch(https://elasticsearch:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"failed to authenticate user [elastic]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}

I replaced password: "${ELASTIC_PASSWORD} with password: mypassword in apm-server.yml and the server started working perfectly. I see services in Kibana/APM, Traces, etc.

Is this an issue with the apm-server.keystore?

[fxdgear]

can you try to docker exec -it apm_server bash after you're in the container try to list the keystore items apm-server keystore list lets see if the password got set on the keystore.

[npapagna]

Hi!

This is what i've got:

bash-4.2$ apm-server keystore list
ELASTIC_PASSWORD

[fxdgear]

@npapagna sorry for such a long delay in a response :(

So it seems like there's an issue with getting the keystore data in the config file for apm-server. I'm glad you have a working workaround. But you've been able to confirm that the keystore is infact storing the password.

I'm going to work on some changes to this to get rid of the requirements of using bind mounts for docker and instead use volumes, which abstracts a bit more away but also makes this way more OS agnostic and allows for better handling of permissions.

[npapagna]

Thanks @fxdgear for taking the time to look at this issue. Sounds good! 👍