[APM] Inject agent config directly into APM Fleet policies

[axw]

As part of the move to Fleet we will need to move away from fetching agent config directly from Kibana, as the privileges APM Server is given do not cover this. Instead, agent config will be pushed down to APM Server via the server's policy.

When agent config is created/updated/removed in Kibana, APM Fleet policies should be updated to include the new agent config directly. For each config block, Kibana will need to supply: the criteria (service name and/or service environment), settings, and an Etag value.

As APM Server will not be communicating directly with Kibana, we will have to come up with a new way of identifying that config has been applied. I can think of two main options, as described in elastic/apm-server#5018:

1. APM Server will index a document whenever a config block is first known to have been applied to an agent. This is similar to what we do today.
2. Agents will periodically send an event to APM Server including agent statistics and the currently applied config Etag, and this will be indexed by the server. This would have the benefit of enabling us to see how many and which agents have applied the config.

Proposed user flow:

1. When superuser visits agent configuration settings, get the list of agent policies -> check each for the apm integration package policy
2. If an APM fleet integration exists, then display a checkbox in agent configuration settings to synchronize settings with fleet management
3. When checked, agent configs will be copied to policy and can only be modified by the super user (other users will see a ready-only page)

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[simitt]

Adding the reference to @axw 's POC here.

[simitt]

Every user with APM app privileges can configure agent central configuration in the APM app. The Fleet app requires superuser privileges for every operation, therefore also for pushing down configuration changes to any APM integrations policy.

Two ways of solving this come to mind:

• Check a users privileges and show a warning in the UI that their changes will only be passed to the APM integrations whenever a superuser logs into Kibana and navigates to the APM config management settings page. OR
• require superuser privileges for changing APM config management settings as soon as at least one Fleet policy with an APM integration is dedected. This is the least favorable, as it would break existing behavior if someone plays around with adding an APM integration.

Potential flow:

• A non-superuser creates/deletes/updates an apm agent central configuration setting. On save the APM app shows a warning that the changes will only be applied to APM integration policies (if any exist) whenever a superuser next navigates to the apm agent central configuration page. There is no way to only show this if an apm integration exists, as for querying this information superuser privileges are required.
• A superuser navigates to the apm agent central configuration UI OR creates/deletes/updates an apm agent central configuration setting. The APM app fetches all existing agent policies that contain an APM integration and updates the complete apm-server.agent_config.* configuration section. The complete section needs to be updated, to also contain the changes potentially made earlier by non-superusers.

Generally, the current apm agent central configuration logic needs to keep being supported as long as running an APM server standalone is supported.