[Filebeat Hints Autodiscover] Fail to merge processors from hints with default_config's processors

[MichaelKatsoulis]

Description

In Filebeat deployed in a Kubernetes cluster a user can select autodiscovery with kubernetes provider instead of using filebeat.inputs. This can be done by commenting the filebeat.inputs block and uncommenting the filebeat.autodiscover one as described in

beats/deploy/kubernetes/filebeat-kubernetes.yaml

Line 125 in 8a70712

# To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:

.
If hints are enabled, in the hints.default_config: block the user specifies which input will be used for each discovered resource and can additionally set processors, parsers and paths. As an example:

  filebeat.yml: |-
    # filebeat.inputs:
    # - type: container
    #   paths:
    #     - /var/log/containers/*.log
    #   processors:
    #     - add_kubernetes_metadata:
    #         host: ${NODE_NAME}
    #         matchers:
    #         - logs_path:
    #             logs_path: "/var/log/containers/"

    To enable hints based autodiscover, remove `filebeat.inputs` configuration and uncomment this:
    filebeat.autodiscover:
     providers:
       - type: kubernetes
         node: ${NODE_NAME}
         hints.enabled: true
         hints.default_config:
           type: container
           paths:
             - /var/log/containers/*${data.kubernetes.container.id}.log
           processors:
            - add_tags:
                tags: [web]
                target: "environment"

User can then deploy a pod and annotate it with hints to include additional processors like this :

cat redis-pod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: redis
  annotations: 
    co.elastic.logs/json.add_error_key: "true"
    co.elastic.logs/processors.add_fields.target: "project"
    co.elastic.logs/processors.add_fields.fields.name: "myproject"
spec:
  containers:
  - name: redis
    image: redis:5.0.4
    command:
      - redis-server
      - "/redis-master/redis.conf"

Expected behaviour

The expected behaviour would be that for Redis discovered pod the container input will be used. Its configuration would include both the processors (add_tags from default_config and add_fields from hints). It should look like this (for processors alone):

processors:
    - add_tags:
        tags: [web]
        target: "environment"
    - add_fields:
        target: project
        fields:
           name: myproject

Result

Instead filebeat logs an error:

{"log.level":"error","@timestamp":"2023-10-13T14:01:46.466Z","log.logger":"autodiscover.cfgfile","log.origin":{"file.name":"cfgfile/list.go","file.line":138},"message":"Error creating runner from config: each processor must have exactly one action, but found 2 actions (add_fields,add_tags)","service.name":"filebeat","ecs.version":"1.6.0"}

Both processors don't work and Redis logs are not collected.

Reason

The reason for that error is that the merged processors configuration looks like this:

processors:
    - add_tags:
        tags: [web]
        target: "environment"
      add_fields:
	      target: project
	      fields:
	        name: myproject

add_tags and add_fields are grouped together under the same processor index. add_fields is not appended even though the processors is a list. This happens in this code line where the hints config is merged to default_config. The Merge method used by default does not append elements to lists. This is explained in this issue

Solution

Instead of the merge method, MergeWithOpts should be used with the option AppendValues.
config.MergeWithOpts(tempCfg, ucfg.AppendValues).
That way processors or parsers or paths found in the hints will be appended in the existing ones from default_config.
This has been tested to work properly.

[MichaelKatsoulis]

cc @ChrsMark , @gizas