Add support for hints' based autodiscovery in kubernetes provider

[ChrsMark]

As discussed at #613 (comment), kubernetes provider can be enhanced in order to support hint's based autodiscovery.

This can be achieved using the already supported dynamic variable resolution mechanism by constructing and emitting hint's specific mappings similar to elastic-agent/internal/pkg/composable/providers/kubernetes/pod.go.

Find below a full example of how we can reach from k8s annotations to an Agent input.

Hint's as annotations

annotations:
  co.elastic.hints/package: redis
  co.elastic.hints/data_streams: info, key
  co.elastic.hints/host: '${kubernetes.pod.ip}:6379'
  co.elastic.hints/info.period: 1m
  co.elastic.hints/key.period: 10m

Emitted mapping by k8s provider

{
  "kubernetes": {
    "hints": {
      "redis": {
        "info": {
          "enabled": true,
          "period": "1m",
          "host": "152.10.67.976:6379"
        },
        "key": {
          "enabled": true,
          "period": "10m",
          "host": "152.10.67.976:6379"
        }
      }
    }
  }
}

Defined template constructed by Fleet UI

providers:
  kubernetes:
    kube_config: /Users/chrismark/.kube/config
    node: "kind-control-plane"
    hints.enabled: true

inputs:
  - name: templates.d/redis/0.3.6
    type: redis/metrics
    data_stream.namespace: default
    use_output: default
    streams:
      - data_stream:
          dataset: redis.info
          type: metrics
        metricsets:
         - info
        hosts:
          - "${kubernetes.hints.redis.info.host|kubernetes.hints.redis.host|'127.0.0.1:6379'}"
        idle_timeout: 20s
        maxconn: 10
        network: tcp
        period: "${kubernetes.hints.redis.info.period|kubernetes.hints.redis.period|'10s'}"
        condition: ${kubernetes.hints.redis.info.enabled} == true or ${kubernetes.hints.redis.enabled} == true //enabled even if only the integration is enabled, AKA enabled by default
      - data_stream:
          dataset: redis.key
          type: metrics
        metricsets:
          - key
        hosts:
          - "${kubernetes.hints.redis.key.host|kubernetes.hints.redis.host|'127.0.0.1'}:${kubernetes.hints.redis.info.port|'6379'}"
        idle_timeout: 20s
        key.patterns:
          - limit: 20
            pattern: '*'
        maxconn: 10
        network: tcp
        period: "${kubernetes.hints.redis.key.period|kubernetes.hints.redis.period|'10s'}"
        condition: ${kubernetes.hints.redis.key.enabled} == true and ${kubernetes.hints.redis.enabled} == true //enabled only if explicitely enabled, AKA disabled by default

The above will be provided as a ConfigMap mounted into the Agent Pod:

apiVersion: v1
kind: ConfigMap
metadata:
  name: elastic-agent-standalone-inputs
data:
  redis.yml: |-
    inputs:
     - name: templates.d/redis/0.3.6
       type: redis/metrics
       data_stream.namespace: default
       use_output: default
       streams:
         - data_stream:
             dataset: redis.info
             type: metrics
           metricsets:
             ...

The scope of this issue is summarised into 2 parts and can be implemented already regardless of Fleet's UI implementation part:

• Update the kubernetes provider accordingly to extract hints' from annotations and produce hints-specific mappings: Add support for hints' based autodiscovery in kubernetes provider #698

• Update the manifests for standalone Agent at https://github.com/elastic/elastic-agent/tree/main/deploy/kubernetes/elastic-agent-standalone to include an inputs.d ConfigMap similar to modules.d of Metricbeat. Provide a dummy inputs.d as an example/placeholder: Update k8s manifests to leverage hints #1202

• Add documentation: Add hints based autodiscover part observability-docs#2182

Known issues/dependencies:

• Inputs.d usage in container environments is fuzzy #663
• : character in dynamic variables breaks the rendering #624

Closes #274

[ChrsMark]

It seems that with Metrics collection the current mechanism could cover our needs without issues with the proper additions. Made some tests to figure out how we will handle container logs and the results are good. My main concern here was about handling the container input providing the option to define stream like what we do in Filebeat with https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#_co_elastic_logsfileset

For the testing I use an apache webserver as the target container and I label it properly so as to collect its logs using the existing kubernetes provider.

Sample ConfigMap for Agent:

agent-configmap.yml

apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-node-datastreams
  namespace: kube-system
  labels:
    k8s-app: elastic-agent-standalone
data:
  agent.yml: |-
    outputs:
      default:
        type: elasticsearch
        hosts:
          - >-
            ${ES_HOST}
        username: ${ES_USERNAME}
        password: ${ES_PASSWORD}
    agent:
      monitoring:
        enabled: true
        use_output: default
        logs: true
        metrics: true
    providers.kubernetes:
      node: ${NODE_NAME}
      scope: node
    inputs:
      - name: apache-1
        type: filestream
        use_output: default
        condition: ${kubernetes.labels.hints} == "parse"
        meta:
          package:
            name: apache
            version: 1.3.5
        data_stream:
          namespace: default
        streams:
          - data_stream:
              dataset: apache.access
              type: logs
            paths:
              - "/var/log/containers/*${kubernetes.container.id}.log"
            tags:
              - apache-access
            exclude_files:
              - .gz$  # Is this needed ???
            prospector.scanner.symlinks: true
            parsers:
                - container:
                  stream: stdout
                  format: auto
          - data_stream:
              dataset: apache.error
              type: logs
            paths:
              - "/var/log/containers/*${kubernetes.container.id}.log"
            exclude_files:
              - .gz$
            tags:
              - apache-error
            processors:
              - add_locale: null
            prospector.scanner.symlinks: true
            parsers:
              - container:
                stream: stderr # Here users would use co.elastic.hints/access.stream: stderr
                format: auto

Sample Apache:

apache.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpd-deployment-xfusion
  labels:
    app: httpd_app_xfusion
spec :
  replicas: 2
  selector:
    matchLabels:
      app: httpd_app_xfusion
  template:
    metadata:
      labels:
        hints: parse
        app: httpd_app_xfusion
    spec :
      containers:
        - name: httpd-container-xfusion
          image: httpd:latest
          ports:
            - containerPort : 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpd-service-xfusion
spec:
  type: ClusterIP
  selector:
    app: httpd_app_xfusion
  ports:
    - port: 8086
      targetPort: 80

Then Elastic Agent starts collecting logs from the proper container stream and sends them to ES.
After installing the assets of the apache integration are manually installed based on https://www.elastic.co/guide/en/fleet/current/install-standalone-elastic-agent.html and https://www.elastic.co/guide/en/fleet/current/install-uninstall-integration-assets.html#install-integration-assets the logs are parsed using the proper pipeline. A sample document ingested is the following:

apache.doc.json

{
  "_index": ".ds-logs-apache.access-default-2022.07.04-000001",
  "_id": "muNSyYEBdZFFax9TmqND",
  "_version": 1,
  "_score": 1,
  "_source": {
    "container": {
      "image": {
        "name": "httpd:latest"
      },
      "runtime": "containerd",
      "id": "ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a"
    },
    "kubernetes": {
      "container": {
        "name": "httpd-container-xfusion"
      },
      "node": {
        "uid": "f6f42049-3f43-427e-8b43-dbd2cecc1f46",
        "hostname": "kind-control-plane",
        "name": "kind-control-plane",
        "labels": {
          "node_kubernetes_io/exclude-from-external-load-balancers": "",
          "node-role_kubernetes_io/master": "",
          "kubernetes_io/hostname": "kind-control-plane",
          "node-role_kubernetes_io/control-plane": "",
          "beta_kubernetes_io/os": "linux",
          "kubernetes_io/arch": "amd64",
          "kubernetes_io/os": "linux",
          "beta_kubernetes_io/arch": "amd64"
        }
      },
      "pod": {
        "uid": "bbb0368f-91fe-4af2-9fda-05503d02bef3",
        "ip": "10.244.0.6",
        "name": "httpd-deployment-xfusion-5546996bfc-48znw"
      },
      "namespace": "default",
      "namespace_uid": "81609706-b8f8-4327-a981-f7a1e0805147",
      "replicaset": {
        "name": "httpd-deployment-xfusion-5546996bfc"
      },
      "namespace_labels": {
        "kubernetes_io/metadata_name": "default"
      },
      "labels": {
        "app": "httpd_app_xfusion",
        "hints": "parse",
        "pod-template-hash": "5546996bfc"
      },
      "deployment": {
        "name": "httpd-deployment-xfusion"
      }
    },
    "agent": {
      "name": "kind-control-plane",
      "id": "ded3358e-655a-45f3-adcf-618b068e90d5",
      "type": "filebeat",
      "ephemeral_id": "c619f972-34de-403d-ac1f-1dc188a952f5",
      "version": "8.3.0"
    },
    "log": {
      "file": {
        "path": "/var/log/containers/httpd-deployment-xfusion-5546996bfc-48znw_default_httpd-container-xfusion-ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a.log"
      },
      "offset": 2188
    },
    "elastic_agent": {
      "id": "ded3358e-655a-45f3-adcf-618b068e90d5",
      "version": "8.3.0",
      "snapshot": false
    },
    "source": {
      "address": "127.0.0.1",
      "ip": "127.0.0.1"
    },
    "apache": {
      "access": {}
    },
    "ecs": {
      "version": "1.12.0"
    },
    "stream": "stdout",
    "host": {
      "hostname": "kind-control-plane",
      "os": {
        "kernel": "4.9.184-linuxkit",
        "codename": "focal",
        "name": "Ubuntu",
        "family": "debian",
        "type": "linux",
        "version": "20.04.4 LTS (Focal Fossa)",
        "platform": "ubuntu"
      },
      "ip": [
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "10.244.0.1",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2",
        "10.244.0.1",
        "10.244.0.1"
      ],
      "containerized": true,
      "name": "kind-control-plane",
      "mac": [
        "02:42:ac:12:00:02",
        "2a:a1:f4:21:45:db",
        "4e:a6:5a:2c:bd:0e",
        "72:0a:5b:02:39:10",
        "b6:54:ec:dc:ae:fd",
        "be:e3:32:16:ff:7e",
        "fa:33:ca:2f:3c:d7"
      ],
      "architecture": "x86_64"
    },
    "event": {
      "agent_id_status": "auth_metadata_missing",
      "ingested": "2022-07-04T13:07:22Z",
      "kind": "event",
      "created": "2022-07-04T13:07:18.842Z",
      "category": "web",
      "dataset": "apache.access",
      "outcome": "success"
    },
    "url": {
      "path": "/",
      "original": "/"
    },
    "tags": [
      "apache-access"
    ],
    "input": {
      "type": "filestream"
    },
    "orchestrator": {
      "cluster": {
        "name": "kind",
        "url": "kind-control-plane:6443"
      }
    },
    "@timestamp": "2022-07-04T13:07:18.000Z",
    "data_stream": {
      "namespace": "default",
      "type": "logs",
      "dataset": "apache.access"
    },
    "http": {
      "request": {
        "method": "GET"
      },
      "response": {
        "status_code": 200,
        "body": {
          "bytes": 45
        }
      },
      "version": "1.1"
    },
    "user": {
      "name": "-"
    }
  },
  "fields": {
    "orchestrator.cluster.name": [
      "kind"
    ],
    "kubernetes.node.uid": [
      "f6f42049-3f43-427e-8b43-dbd2cecc1f46"
    ],
    "event.category": [
      "web"
    ],
    "elastic_agent.version": [
      "8.3.0"
    ],
    "kubernetes.namespace_uid": [
      "81609706-b8f8-4327-a981-f7a1e0805147"
    ],
    "kubernetes.deployment.name": [
      "httpd-deployment-xfusion"
    ],
    "host.os.name.text": [
      "Ubuntu"
    ],
    "host.hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.kubernetes_io/os": [
      "linux"
    ],
    "host.mac": [
      "02:42:ac:12:00:02",
      "2a:a1:f4:21:45:db",
      "4e:a6:5a:2c:bd:0e",
      "72:0a:5b:02:39:10",
      "b6:54:ec:dc:ae:fd",
      "be:e3:32:16:ff:7e",
      "fa:33:ca:2f:3c:d7"
    ],
    "container.id": [
      "ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/master": [
      ""
    ],
    "kubernetes.labels.pod-template-hash": [
      "5546996bfc"
    ],
    "container.image.name": [
      "httpd:latest"
    ],
    "http.request.method": [
      "GET"
    ],
    "kubernetes.labels.app": [
      "httpd_app_xfusion"
    ],
    "host.os.version": [
      "20.04.4 LTS (Focal Fossa)"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/os": [
      "linux"
    ],
    "kubernetes.namespace": [
      "default"
    ],
    "host.os.name": [
      "Ubuntu"
    ],
    "source.ip": [
      "127.0.0.1"
    ],
    "agent.name": [
      "kind-control-plane"
    ],
    "host.name": [
      "kind-control-plane"
    ],
    "event.agent_id_status": [
      "auth_metadata_missing"
    ],
    "http.response.status_code": [
      200
    ],
    "http.version": [
      "1.1"
    ],
    "event.kind": [
      "event"
    ],
    "event.outcome": [
      "success"
    ],
    "host.os.type": [
      "linux"
    ],
    "input.type": [
      "filestream"
    ],
    "log.offset": [
      2188
    ],
    "data_stream.type": [
      "logs"
    ],
    "tags": [
      "apache-access"
    ],
    "host.architecture": [
      "x86_64"
    ],
    "container.runtime": [
      "containerd"
    ],
    "url.path": [
      "/"
    ],
    "agent.id": [
      "ded3358e-655a-45f3-adcf-618b068e90d5"
    ],
    "host.containerized": [
      true
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "event.created": [
      "2022-07-04T13:07:18.842Z"
    ],
    "kubernetes.node.labels.node-role_kubernetes_io/control-plane": [
      ""
    ],
    "agent.version": [
      "8.3.0"
    ],
    "host.os.family": [
      "debian"
    ],
    "kubernetes.node.name": [
      "kind-control-plane"
    ],
    "kubernetes.node.hostname": [
      "kind-control-plane"
    ],
    "user.name": [
      "-"
    ],
    "kubernetes.pod.uid": [
      "bbb0368f-91fe-4af2-9fda-05503d02bef3"
    ],
    "source.address": [
      "127.0.0.1"
    ],
    "host.ip": [
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "10.244.0.1",
      "172.18.0.2",
      "fc00:f853:ccd:e793::2",
      "fe80::42:acff:fe12:2",
      "10.244.0.1",
      "10.244.0.1"
    ],
    "orchestrator.cluster.url": [
      "kind-control-plane:6443"
    ],
    "agent.type": [
      "filebeat"
    ],
    "event.module": [
      "apache"
    ],
    "stream": [
      "stdout"
    ],
    "host.os.kernel": [
      "4.9.184-linuxkit"
    ],
    "kubernetes.pod.name": [
      "httpd-deployment-xfusion-5546996bfc-48znw"
    ],
    "elastic_agent.snapshot": [
      false
    ],
    "kubernetes.pod.ip": [
      "10.244.0.6"
    ],
    "kubernetes.container.name": [
      "httpd-container-xfusion"
    ],
    "elastic_agent.id": [
      "ded3358e-655a-45f3-adcf-618b068e90d5"
    ],
    "data_stream.namespace": [
      "default"
    ],
    "kubernetes.replicaset.name": [
      "httpd-deployment-xfusion-5546996bfc"
    ],
    "host.os.codename": [
      "focal"
    ],
    "kubernetes.namespace_labels.kubernetes_io/metadata_name": [
      "default"
    ],
    "http.response.body.bytes": [
      45
    ],
    "kubernetes.node.labels.kubernetes_io/hostname": [
      "kind-control-plane"
    ],
    "kubernetes.node.labels.beta_kubernetes_io/arch": [
      "amd64"
    ],
    "event.ingested": [
      "2022-07-04T13:07:22.000Z"
    ],
    "url.original": [
      "/"
    ],
    "kubernetes.labels.hints": [
      "parse"
    ],
    "@timestamp": [
      "2022-07-04T13:07:18.000Z"
    ],
    "host.os.platform": [
      "ubuntu"
    ],
    "log.file.path": [
      "/var/log/containers/httpd-deployment-xfusion-5546996bfc-48znw_default_httpd-container-xfusion-ea36b297cd0d1e576eb358f93ec7a2489b9741daeea1ca0af233053b1f6b529a.log"
    ],
    "data_stream.dataset": [
      "apache.access"
    ],
    "kubernetes.node.labels.kubernetes_io/arch": [
      "amd64"
    ],
    "agent.ephemeral_id": [
      "c619f972-34de-403d-ac1f-1dc188a952f5"
    ],
    "kubernetes.node.labels.node_kubernetes_io/exclude-from-external-load-balancers": [
      ""
    ],
    "event.dataset": [
      "apache.access"
    ]
  }
}

So it seems that for logs we need something like the following in the templates:

  - name: apache-1
    type: filestream
    use_output: default
    meta:
      package:
        name: apache
        version: 1.3.5
    data_stream:
      namespace: default
    streams:
      - data_stream:
          dataset: apache.access
          type: logs
        paths:
          - "/var/log/containers/*${hints.container.id}.log"
        tags:
          - apache-access
        exclude_files:
          - .gz$  # Is this needed ???
        prospector.scanner.symlinks: true
        parsers:
            - container:
              stream: "${hints.redis.access.stream|'all'}" # Here users would use co.elastic.hints/access.stream: stdout
              format: auto
      - data_stream:
          dataset: apache.error
          type: logs
        paths:
          - "/var/log/containers/*${hints.container.id}.log"
        exclude_files:
          - .gz$
        tags:
          - apache-error
        processors:
          - add_locale: null
        prospector.scanner.symlinks: true
        parsers:
          - container:
            stream: "${hints.redis.error.stream|'all'}" # Here users would use co.elastic.hints/access.stream: stderr
            format: auto

Note that we add the container parser and we set the path accordingly.

@gizas @ruflin let me know what you think.

[ChrsMark]

At the moment inputs with defined processors will not be able to receive extra processors coming from the hints because of a known limitation: #735