Selective monitoring of Kubernetes workloads using annotations for agent standalone

[mlunadia]

Also known as hints based auto-discovery for agent standalone
Picking up context from elastic/beats#23876

We want to enable users to monitor Kubernetes workloads from the resource side for users of agent standalone.

User outcome
Kubernetes users can declare their workloads and include hints, similar to those described in beats documentation metricbeat and filebeat manifests and then expect agent whilst enrolled in standalone mode to pick these hints launch the proper config for it.

Acceptance criteria
On top of the outcome described above

• Application operator (developer) should have an accessible way to find the list of available hints
• Application operator (developer) should receive feedback for when a hint is not valid

Implementation issues

• Add support for template construction compatible with kubernetes hints: Add support for template construction compatible with kubernetes hints kibana#135624 will be implemented in CI jobs
• Add support for hints' based autodiscovery in kubernetes provider #662
• Documentation issue: Add hints based autodiscover part observability-docs#2182

[ChrsMark]

We had an initial chat about this with @gizas and in order to start pushing this forward we first need to decide on the approach we will follow to make this feature available to our users. Keep in mind that this feature requires cross team effort from Cloudnative, Agent and Fleet UI teams. The implementation can happen from one or more teams accordingly but first we need to have a solid plan about the way forward. Once we have decided the what and the how we can create smaller, detail specific issues for the various parts of the feature's implementation. Here is a first outline of the current status of the issue and a high level proposal for its implementation across the "stack".

Hint based autodiscovery in standalone Agent

Related issues from the past

[1] Hints based autodiscovery
[2] Templates support

Current Status

What we have so far?

There is available the implementation to watch for Pods and collect metadata for them.
We can easily identify Pods (Nodes etc) and we can easily identify the hints out of the Resources' annotations.

What we need

We need a mechanism to receive those hints and match them with inputs/integrations. This mechanism can either "live" in Agent to cover standalone mode or in Fleet UI OR in an external component like an additional "operator". In this proposal we only focus on standalone mode.

Standalone mode

The Agent needs to know how packages/inputs look like in order to match them with the hints. One way to achieve this is by using input templates. This would make the flow quite similar to Beats.

Feature design proposal (technical/high-level)

As it was already mentioned, Agent can identify Pods (and other workloads) in Kubernetes and communicate this information to Agent's controller so as to enable and populate inputs' configuration accordingly.
The current proposal is based in the template's support which was discussed already in the past.
The proposed way to add hint's support is the following:

1. Identify hints in Pods' annotations and emit the "hints' event" at the same time the provider emits the mappings for the composable controller at

   elastic-agent/internal/pkg/composable/providers/kubernetes/pod.go

   Line 155 in 7db7406

   _ = p.comm.AddOrUpdate(data.uid, PodPriority, data.mapping, data.processors)

   . We can use a different channel to emit these events or the same but this is an implementation detail to be decided later.
2. Agent's composable controller (or a new kind of controller) receives the hints and tries to utilise them in order to populate the values of the respective template which would be available in data/templates.d. This functionality was outlined at [Elastic Agent] Support for input templates beats#24054 (comment). We can figure out the details accordingly but this is the high level idea.
3. After the successful evaluation of the hints and the input configuration construction Agent adds the new configuration block into the local policy and the input is enabled.

How templates are created

One important "detail" that needs to be discussed is how the templates would become available in Elastic Agent locally. At elastic/beats#24054 (comment) (Part 2) it was proposed to ship the templates along with the Agent binary which implies that we would include them in the build/packaging stage after retrieving them from the Package Registry. This could work but we come with an easier solution (kudos to @gizas :D) which proposes to leave this work to our users and give them the option to download the templates from Fleet UI in the same was they do for the standalone policy. So Fleet UI can create a templates.d.tar on the fly using the latest available packages and then users can extract this zipped directory at data/templates.d.
This gives us the opportunity to make the delivery of the templates easier using the latest versions of the available packages from Fleet UI, and at the same time we follow similar UX with the one we have for the standalone policy. It would be Fleet UI's responsibility to collect the packages and construct the templates to make them available to the users.

User journey

Having described the high level of the technical part we need to also describe how users will us the feature. Here is an example user journey:

1. Users download Elastic Agent binary locally to run in standalone mode. The process is the same with today (https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html).
2. In the next step users would need to create and download the policy as explained at https://www.elastic.co/guide/en/fleet/current/create-standalone-agent-policy.html
3. Final step is that users download the templates from Fleet UI the same way they do for step2
   

** The UX details/wording etc can be improved but this is the high level idea.

Proposed implementation issues

1. Implement hints' identification mechanism in k8s provider side and make hints' events available to Agent's controller/core.

a. specify the list of supported hints
b. lookup for supported hints in the Pod's annotations
c. emit a "hints's event" to a specific channel to be handled by codebase described at issue no2.

code repo: https://github.com/elastic/elastic-agent/tree/7db7406cc04d6dd8f2d47f8a37e299da4598056b/internal/pkg/composable/providers/kubernetes
maintainer team: cloudnative-monitoring

2. Implement hints-to-inputs logic based on input templates (based on item 2)

a. Define template schema. Provide a sample.
b. Implement support for input templates in Agent. This is actually what was proposed with elastic/beats#24054 but it seems it was closed after some time.
c. Implement the linking codebase that would receive hints events (item 1) and match them with the proper template. Then enable the respective input.

code repo: https://github.com/elastic/elastic-agent/tree/7db7406cc04d6dd8f2d47f8a37e299da4598056b/internal/pkg/composable
maintainer team: Elastic Agent

3. Implement template construction out of the packages on Fleet UI and make them available to users in the same way the standalone policy is provided

code repo: https://github.com/elastic/kibana/tree/main/x-pack/plugins/fleet
maintainer team: Fleet UI

@gizas @mlunadia @ruflin let me know what you think about this and how we could move this forward.

[ruflin]

The nice part about the proposed solution is that step 1 and 2 can happen fully independent from Fleet. Also having support for input templates in Elastic Agent means it does not only support templates coming from Fleet but also users could build their own if needed.

Also if later on we have an orchestrator solution, the templates will still be useful.

We should consider to bundle some templates like system by default with the Elastic Agent for convenience.

An additional argument for this approach is that it does not only work with k8s but also any other standalone setup.

[ph]

As pointed out in elastic/beats#24054 (comment), looking at the described workflow suggested in the comment.
Downloading the package or embedding will works. One thing I believe we are missing is a way to add a runtime constraint on the package itself.
IE you can only run this package with Elastic Agent version 8.5+ or 8.6 and lower.

Concerning the download from the UI we could instead make the agent policy bigger with the template embedded in it, something like this.

inputs:
- type: filestream
- type: tcp

# Instead of looking up on disk we can look at this instead, which could be a VFS representation of the structure on disk.
template.d:
 nginx/1.8:
    ....

The user only have one thing to download and deploy, the yaml is bigger but this reduce the risk from having mistmatch.

[ChrsMark]

Thanks for your reviews @ruflin and @ph :)

Heads-up on this. After the initial feedback and some more brainstorming around this, it seems that for Hints' feature implementation we don't necessarily need the template support.
This implies that we can skip the no2 and no3 items from the Proposed implementation issues list posted above.

The rational behind a such solution is that we can just re-use the k8s provider to emit specific mappings (hints specific ones) that would then be capable to populate the proper “templates” produced by the Fleet UI with the proper conventions.

Let me put some examples together to illustrate the point:

Updated proposal

"Templates" format

We take for granted that templates are constructed by Fleet UI, using the available packages' definitions, and are placed in the elastic-agent.yml as @ph mentioned above.
The difference is that those "templates" would not be listed in a different level but can be listed under inputs level, while being annotated with the proper hints's based conditions. Having these inputs/templates populated by fleet UI then Agent can just leverage the "dynamic variable resolution" capabilities and the "kubernetes provider" to enable those inputs/templates if the proper hints are provided.

For simplicity we can add only the latest packages in the list of templates but we can simply extend this while using the proper conditions accordingly.

Find below a "template sample":

inputs:
...
- name: templates.d/redis/0.3.6
  type: redis/metrics
  use_output: default
  # condition: "${docker.hints.redis.version|kubernetes.hints.redis.version == "0.3.6"}"
  meta:
  package:
    name: redis
    version: 0.3.6
  data_stream:
  namespace: default
  streams:
  - data_stream:
      dataset: redis.info
      type: metrics
    metricsets:
      - info
    hosts:
      - "${docker.hints.redis.info.host|kubernetes.hints.redis.info.host|'127.0.0.1:6379'}"
    idle_timeout: 20s
    maxconn: 10
    network: tcp
    period: "${docker.hints.redis.info.period|kubernetes.hints.redis.info.period|'10s'}"
    condition: "${docker.hints.redis.info.enabled|kubernetes.hints.redis.info.enabled|true}"
   - data_stream:
       dataset: redis.key
       type: metrics
    metricsets:
      - key
    hosts:
      - "${docker.hints.redis.key.host|kubernetes.hints.redis.key.host|'127.0.0.1:6379'}"
    idle_timeout: 20s
    key.patterns:
      - limit: 20
        pattern: '*'
    maxconn: 10
    network: tcp
    period: "${docker.hints.redis.key.period|kubernetes.hints.redis.key.period|'10s'}"
    condition: "${docker.hints.redis.key.enabled|kubernetes.hints.redis.key.enabled|false}"

For now we can follow a simple approach and support only a few "hints" similarly to what we do in Metricbeat/Filebeat. In the above example I only leverage "period" and "host" hints but we can extend it as much as we think.

Note that the conditions per data_stream are suffixed with an |true if the data_stream is enabled by default or |false if the data_stream is disabled by default. This gives us the option to explicitly enable/disable data_streams using the hints or enabling just the integration implying to enable the default data_streams.

Hints' specific mappings

Having defined the "templates" as above we can now leverage the kubernetes provider in order to populate those with the respective values as settings and enable them.

So in the kubernetes provider we can check for specific annotations like co.elastic.hints/*.

A Redis Pod would be annotated like this:

annotations:
  co.elastic.hints/package: redis
  co.elastic.hints/data_streams: info, key
  co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
  co.elastic.hints/period: 1m

Note that the settings can also be data_stream specific if users want it:

annotations:
  co.elastic.hints/package: redis
  co.elastic.hints/data_streams: info, key
  co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
  co.elastic.hints/info.period: 1m
  co.elastic.hints/key.period: 10m

If settings are not data_stream specific then common settings will be used across the data_streams.

With the above annotations the kubernetes provider would produce the following mapping:

{
  "kubernetes": {
    "hints": {
      "redis": {
        "info": {
          "enabled": true,
          "period": "1m",
          "host": "152.10.67.976:6379"
        },
        "key": {
          "enabled": true,
          "period": "10m",
          "host": "152.10.67.976:6379"
        }
      }
    }
  }
}

Note that host field can be populated by the kubernetes provider on the fly since the kubernetes provider already holds the Pod's metadata at the same place.

Having the above mapping we can emit this at

elastic-agent/internal/pkg/composable/providers/kubernetes/pod.go

Line 155 in a950518

_ = p.comm.AddOrUpdate(data.uid, PodPriority, data.mapping, data.processors)

and this would be enough to enable the respective "template" provided in the standalone configuration.

The rendered input configuration will be the following:

inputs:
...
- name: templates.d/redis/0.3.6
  type: redis/metrics
  use_output: default
  meta:
  package:
    name: redis
    version: 0.3.6
  data_stream:
  namespace: default
  streams:
  - data_stream:
      dataset: redis.info
      type: metrics
    metricsets:
      - info
    hosts:
      - "152.10.67.976:6379"
    idle_timeout: 20s
    maxconn: 10
    network: tcp
    period: 1m
   - data_stream:
       dataset: redis.key
       type: metrics
    metricsets:
      - key
    hosts:
      - "152.10.67.976:6379"
    idle_timeout: 20s
    key.patterns:
      - limit: 20
        pattern: '*'
    maxconn: 10
    network: tcp
    period: 10m

Additional considerations

1. One major issue here is how we would install the assets like dashboards and pipelines but this is an issue that is not hints specific but is generic when one run standalone agent. Nevertheless, this is mentioned and documented accordingly at https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html and https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html#install-integration-assets.
2. We should also think about if another component should be responsible for constructing the "templates". For example elastic-package tool could provide this functionality which would allow advanced users to use it directly and construct the templates on their ends. This would also allow elastic-agent to re-use the tool and construct the templates at build time in order to include them in the final distrubution. However I'm leaning towards having Fleet UI to construct these templates and embed them in the elastic-agent.yml policy configuration. This makes things easier and would provide a seamless UX since users are already directed to download the standalone policy through Fleet UI. But we can discuss more about this if there are different opinions.

Since we are considering the templates' construction in Fleet UI, @jen-huang could you chime in and provide feedback on this and more specifically on "Templates" construction by Fleet UI part?

[ph]

@kpollich If you want to provide feed from the Fleet side.

[ruflin]

I like that the above approach is allowing us to use mostly existing mechanism. To make it as easy as possible for users to update their "redis" template for example, it would be great to have an inputs.d/redis-036.yml file or similar. When downloading a new template from Fleet, a user would not have to figure out where their input lists for redis starts or stops, but just replacing the file.

@ph I know we have discussed inputs.d support for Elastic Agent in the past. Did this ever happen?

[ChrsMark]

@ruflin as far as I can see inputs.d are already supported because of 3c2072d, so we are good here 🙂 .

This seems to be more structured but compared to @ph 's suggestion to include everything in elastic-agent.yml standalone policy might be more error prone since we leave a lot of flexibility to users. Including everything in the standalone policy would "hide" everything from the users and allow us to handle it as an implementation detail for our purposes.

However could you elaborate more on how a user journey would look like in that case? I guess users would need to download the templates (in batch) from Fleet UI in a second step after the standalone policy and extract them into the inputs.d directory?

Something that we miss so far is that in Kubernetes everything can be quite packed before deployment time. For example in order to deploy Elastic Agent (standalone) in k8s users would use https://github.com/elastic/elastic-agent/blob/main/deploy/kubernetes/elastic-agent-standalone-kubernetes.yaml following the docs, or they can use the recently added UX added in Fleet UI with elastic/kibana#114439.

For us this would mean that we have several options here:

1. We can easily include the templates in a ConfigMap which will be mounted inside the Agent's Pod. This is exactly the same we do with Beats for example at https://github.com/elastic/beats/blob/main/deploy/kubernetes/metricbeat/metricbeat-daemonset-configmap.yaml#L79
   which is then mounted in Metrcibeat's Pod at https://github.com/elastic/beats/blob/main/deploy/kubernetes/metricbeat-kubernetes.yaml#L198. Similarly to modules.d directory we will have inputs.d.
2. The way this ConfigMap will be produced can be through Fleet UI similarly to what we have in order to produce the k8s manifest from Fleet UI-> In case of kubernetes integration detected return manifest in standalone agent layout instead of policy kibana#114439

My approach here would be that https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone manifests are shipped with a dummy inputs.d ConfigMap (or even an empty one) and then users are directed to download the inputs.d confgiMap from the Fleet UI page and replace it accordingly.
With this approach k8s users/operators can even downloading the inputs.d ConfigMap from time to time and updating it accordingly in order to push it to their repos for version controlling purposes.
There is a lot of flexibility here on how the UX could look like but we can start with sth and iterate accordingly (cc: @mlunadia @gizas)

Having said this I see the following implementation parts:

1. Update the kubernetes provider accordingly to extract hints' from annotations and produce hints-specific mappings as described at Selective monitoring of Kubernetes workloads using annotations for agent standalone #613 (@elastic/obs-cloudnative-monitoring can take care of this)
2. Update the manifests for standalone Agent at https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone to include an inputs.d ConfigMap similar to modules.d of Metricbeat. Provide a dummy inputs.d as an example/placeholder.(@elastic/obs-cloudnative-monitoring can take care of this)
3. Update Fleet UI accordingly to produce the inputs.d ConfigMap that will then be placed inside the elastic-agent-standalone/elastic-agent-standalone-daemonset-configmap.yaml

[ruflin]

However could you elaborate more on how a user journey would look like in that case? I guess users would need to download the templates (in batch) from Fleet UI in a second step after the standalone policy and extract them into the inputs.d directory?

You have a good point here. It is really neat that we can offer the user a single yml file to download and it just works. For the Elastic Agent configs, I started to think about separating output configs from input configs. Output config in the standalone case rarely change, inputs much more often and are likely managed by the user eventually in Git or similar. Could still have a single yaml but with multiple sections inside? Have some questions around this, will ping you directly.

[mlunadia]

@ChrsMark looking at the user journey, how do we satisfy the two items below?

Acceptance criteria
On top of the outcome described above

• Application operator (developer) should have an accessible way to find the list of available hints
• Application operator (developer) should receive feedback for when a hint is not valid

[ChrsMark]

@ChrsMark looking at the user journey, how do we satisfy the two items below?

Acceptance criteria
On top of the outcome described above

• Application operator (developer) should have an accessible way to find the list of available hints
• Application operator (developer) should receive feedback for when a hint is not valid

Quick answer is:

1. docs like at https://www.elastic.co/guide/en/beats/metricbeat/current/configuration-autodiscover-hints.html
2. logs of Agent. As much verbose as we want

Beyond this we can go quite further and maybe provide some extra troubleshooting functionality with elastic-agent inspect command.

However I think these should not affect the base implementation we are discussing here since are mostly enhancements on top of the core implementation?

[ph]

@ruflin Yes we do have support for input.d, but as I mentioned having the possibility of making the configuration atomic and a single thing to deploy make sense to reduce the possible error to ensure that the right thing at the right version is deployed.

We can support both worklow without too much trouble the IAC and a more click and get.

@ChrsMark I am +1 for the proposal above.

Concerning 2, we might want to cleanup the log to make sure we reduce the noise to problem ratio.