[Ingest] Fleet should allow to enable elastic-agent monitoring

[nchaulet]

Description

We should allow the user to configure elastic-agent monitoring, I think this should be done in the configuration pages?

Are we okay with using the default output for monitoring for alpha1? @ph

We should be able to add this to the generated config

outputs:
  default:
    type: elasticsearch
    api_key: VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw
    hosts: ["localhost:9200"]
    ca_sha256: "7lHLiyp4J8m9kw38SJ7SURJP4bXRZv/BNxyyXkCcE/M="
    # Not supported at first
    queue:
      type: disk

  long_term_storage:
    type: elasticsearch
    api_key: VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw
    hosts: ["localhost:9200"]
    ca_sha256: "7lHLiyp4J8m9kw38SJ7SURJP4bXRZv/BNxyyXkCcE/M="
    queue:
      type: disk

  monitoring:
    type: elasticsearch
    api_key: VuaCfGcBCdbkQm-e5aOx:ui2lp2axTNmsyakw9tvNnw
    hosts: ["localhost:9200"]
    ca_sha256: "7lHLiyp4J8m9kw38SJ7SURJP4bXRZv/BNxyyXkCcE/M="

settings.monitoring:
  use_output: monitoring

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@nchaulet yes, we are OK with that, for the first release.

[ph]

@nchaulet would they have the same credentials and permissions as the default output?

[nchaulet]

@hbharding we need to be able to enable agent monitoring when we create/update an agent config, do you have anything in mind on how we show that (copy)?

I implemented it like this for now under advanced options

[nchaulet]

@ph unless you see a reason to not, yes it will be the same credentials and permissions.

[nchaulet]

@ph I am trying to test it locally does the agent monitoring is already implemented agent side? and should I be able to see metrics or anything in kibana?
I am sending this config

id: 6d7a4780-7f21-11ea-b609-3923382c3c4a
revision: 1
settings.monitoring:
  use_output: default
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'http://localhost:9200'
    api_key: 'REDACTED'
datasources: []

[ph]

@michalpristas can you help @nchaulet on this?

[michalpristas]

to enable monitoring you need to specify this:

settings.monitoring:
  # enabled turns on monitoring of running processes
  enabled: true
  # enables log monitoring
  logs: true
  # enables metrics monitoring
  metrics: true

it will look for use_output, this is correct, this needs to be of type elasticsearch.

Specifying output is not enough to get this working, you need to enable monitoring explicitly

[nchaulet]

@michalpristas Do you think it make sense to allow user to enable just logs or metrics from kibana? or we should always enable everything?

[michalpristas]

i think it makes sense, in some way user has this behavior now as we provide filebeat and metricbeat searately. programmatically it was a natural way as these are separate entities.
from user perspective we should double check with @ruflin or @mostlyjason

[mostlyjason]

What do you think about making Agent monitoring an integration package? It could be included by default similar to the system package, but the user could use our existing data source editor to control the settings. We could also offer assets like dashboards, alerts, etc.