Fleet managed: enable debug mode

[mtojek]

Hi Team,

while investigating the root cause of elastic/integrations#1566 , we confirmed that's really useful to enable debug logs for filebeat and metricbeat running as Docker containers (under CI). The ideal option would involve an extra property in policy to enable debug logging or at least a special Docker image ENV which can be hardcoded in system tests.

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[mtojek]

Hey! Is there an option to give some priority? It might be a useful feature for supporting customers once we're GA. Any workaround would be also appreciated.

[jlind23]

Assigning @michel-laterman to have more insight as he worked on the diagnosis topic. As soon as we have a clear target on it we'll move forward.
(fyi @nimarezainia )

[michel-laterman]

We can pass the logging.level setting to beats/processes

[jlind23]

Should we add it in the same PR as ECS logging? elastic/beats#28573
wdyt @michel-laterman

[mtojek]

We can pass the logging.level setting to beats/processes

Sounds good, but to be complete we need to control the log level with ENVs injected to the Docker image. Is this option available?

[michel-laterman]

There's some support for env vars now (https://www.elastic.co/guide/en/beats/metricbeat/7.15/using-environ-vars.html) however it needs changes made to the config file used in order to work, is that acceptable for the time being?

[mtojek]

This is the way we start elastic-agent and fleet-server:
https://github.com/elastic/elastic-package/blob/9e5e39f0bfcdd0d92439384b4e86836da47b1391/internal/profile/_static/docker-compose-stack.yml#L94

We depend on environment variables only.

[michel-laterman]

After a chat with @ruflin, I think that we agreed the best approach to take was to add the ability to define logging levels to the policy instead of just passing the agent's level to the process. This way we can use environment variables with a default, for example (syntax may not be correct)

log.level: ${CUSTOM_LOG_LEVEL:'info'}

[ruflin]

@joshdover I wonder if we should use an environment variable with the preset values as the default from Fleet. This means, the policy sent down by Fleet would always contain something like log.level: ${env.ELASTIC_AGENT_LOG_LEVEL|'error'} in the case of error (syntax may also not be correct).

Now what happens if a user was overwriting the log level for an elastic agent from Fleet? Which one wins?