separate "bootstrap" from retrieved config when running in agent mode

[michel-laterman]

Follow up for #3411

fleet-server has been altered so that in agent-mode the policy self-monitor can send partial config updates to change output settings based off the output section of the fleet-server policy.

However this can introduce an edge case when the server is running where a policy-change unrelated to the output will result in the "bootstrap" output sent from the elastic-agent being used temporarily.
The sequence of events may be:

• ES hosts are set to A, B
• fleet-server is enrolled with host A
  • policy monitor will send A, B to reload
• fleet-server policy is adjusted by user (i.e., max_agent has changed)
• elastic-agent will send updated policy with only host A (if the agent has not been restarted)
  • policy monitor will then sent A, B to reload again

The issue here is the final reload where the fleet-server because the reload operation will restart the API server.

When receiving new output fleet-server may need to determine how to insert/inject output (and the revision idx value) into the config so that the output from policy does not trigger a second reload.

[cmacknz]

I think the root of our problems is that agent overwrites the outputs from the policy with the bootstrap output. This makes the bootstrap process less complex from Fleet Server's perspective.

We need agent to always pass the outputs from the policy and just augment them with the bootstrap output, because there is no guarantee that the initial bootstrap output remains valid over the complete life of fleet server since it is set only at install time.

We could include the bootstrap output as a second output unit separate from the ones in the agent policy. Agent would always inject it so that Fleet server always receives two output units.

Another options if adding a second output unit is too complex is to create a bootstrap: sub-section of the existing output block that contains the bootstrap output. Something like:

output:
  elasticsearch:
    hosts: '${ELASTICSEARCH_HOSTS:https://localhost:9200}'
    service_token: '${ELASTICSEARCH_SERVICE_TOKEN}'
    ssl.ca_trusted_fingerprint: '${ELASTICSEARCH_CA_TRUSTED_FINGERPRINT}'
    bootstrap:
      hosts: 'https://bootstrap:9200'
      service_token: '${ELASTICSEARCH_SERVICE_TOKEN}'
      ssl.ca_trusted_fingerprint: '${ELASTICSEARCH_CA_TRUSTED_FINGERPRINT}'

[cmacknz]

We need agent to always pass the outputs from the policy and just augment them with the bootstrap output, because there is no guarantee that the initial bootstrap output remains valid over the complete life of fleet server since it is set only at install time.

Having agent always pass the outputs from the policy works around the edge case in the description, because agent always has a cached copy of the most recent policy on disk. If the bootstrap output is offline for whatever reason, Fleet server is still going to receive the other outputs from this local persisted copy of the policy from the agent.

[michel-laterman]

Note that reloading output from policy has been reverted: #3495
We'll follow up with a better testing definition for this item