[BUG] [parser /container] Error with Reduced max_log_size in OpenTelemetry Collector's Filelog Receiver

[Masmiiadm]

Component(s)

pkg/stanza

What happened?

Description

I based the new log parsing setup for the OpenTelemetry Collector on this documentation, and everything worked perfectly with the default configuration.

However, when I tried reducing the max_log_size parameter in the filelogreceiver configuration from its default value of 1 MiB to 40 KiB, I encountered the following error
"data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log", "error": "failed to detect a valid container log format: entry cannot be parsed as container logs" "deployment\": {\".\": {}, \"f:instanceName\": {}, \"f:namespace\": {}}, \"f:environment\": {\".\": {}, \"f:isKubernetes\": {}}, \"f....

Expected Result

the fileconsumer parses these logs successfully like the 1MiB default configuration

Actual Result

the filelog receiver and th container parser failed to detect a valid container log format when configure max_log_size 40 KiB

Collector version

0.111.0

Environment information

No response

OpenTelemetry Collector configuration

apiVersion: v1
data:
  relay: |
    exporters:
      debug: {}
    extensions:
      health_check:
        endpoint: ${env:MY_POD_IP}:13133
    processors:
      batch: {}
      k8sattributes:
        extract:
          metadata:
          - k8s.namespace.name
          - k8s.deployment.name
          - k8s.statefulset.name
          - k8s.daemonset.name
          - k8s.cronjob.name
          - k8s.job.name
          - k8s.node.name
          - k8s.pod.name
          - k8s.pod.uid
          - k8s.pod.start_time
        filter:
          node_from_env_var: K8S_NODE_NAME
        passthrough: false
        pod_association:
        - sources:
          - from: resource_attribute
            name: k8s.pod.ip
        - sources:
          - from: resource_attribute
            name: k8s.pod.uid
        - sources:
          - from: connection
      memory_limiter:
        check_interval: 5s
        limit_percentage: 80
        spike_limit_percentage: 25
    receivers:
      filelog:
        exclude:
        - /var/log/pods/*/opentelemetry-collector/*.log
        include:
        - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        max_log_size: 40KiB
        operators:
        - id: container-parser
          type: container
        retry_on_failure:
          enabled: true
        start_at: end
    service:
      extensions:
      - health_check
      pipelines:
        logs:
          exporters:
          - debug
          processors:
          - transform
          - k8sattributes
          - memory_limiter
          - resource
          - batch
          receivers:
          - filelog
      telemetry:
        logs:
          level: info
        metrics:
          address: ${env:MY_POD_IP}:8888

Log output

2024-10-28T10:46:19.065Z  error reader/reader.go:148  process: %w {"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log", "error": "failed to detect a valid container log format: entry cannot be parsed as container logs: [\\\"openid\\\", \\\"profile\\\", \\\"email\\\"], \\\"username_claim\\\": \\\"sub\\\"}, \\\"openshift\\\": {\\\"client_id_prefix\\\": \\\"kiali\\\"}, \\\"strategy\\\": \\\"anonymous\\\"}, \\\"custom_dashboards\\\": [], \\\"deployment\\\": {\\\"accessible_namespaces\\\": [\\\"**\\\"], \\\"affinity\\\": {\\\"node\\\": {}, \\\"pod\\\": {}, \\\"pod_anti\\\": {}}, \\\"hpa\\\":

Additional context

No response

[github-actions]

Pinging code owners:

• pkg/stanza: @djaglowski

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[djaglowski]

cc @ChrsMark

[ChrsMark]

Hey @Masmiiadm, could you share a sample log line that causes that failure? From what I can see in the Log Output section of the issue, the entry does not really look like a valid format? Or what I see is the trimmed log line?

I assume that when the max_log_size is set the log lines can be trimmed. In that case the container format might have been corrupted and seeing the parser failing actually makes sense.

[Masmiiadm]

Hi @ChrsMark . you will find the file 0.log attached
0.log

[ChrsMark]

Hi @ChrsMark . you will find the file 0.log attached 0.log

The link does not work for me. Just a single log line in a comment will be fine, no need to share the whole log file :).

[Masmiiadm]

@ChrsMark , bellow the log line , thx
' {"level":"error","ts":1730728178.0375009,"logger":"runner","msg":"Using /etc/ansible/ansible.cfg as config file\r\n\r\nPLAY [localhost] ******************************************************************************************************\n\r\nTASK [debug] *******************************************************************\nok: [localhost] => {\r\n "msg": "KIALI RECONCILIATION START"\r\n}\n\r\nTASK [debug] *******************************************************************\nok: [localhost] => {\r\n "msg": "CR: name=kiali, namespace=kiali-operator"\r\n}\n\r\nTASK [Playbook start time] *****************************************************\nok: [localhost] => {"ansible_facts": {"playbook_time_start": "2024-11-04 13:49:13"}, "changed": false}\n\r\nTASK [Determine the default playbook] ******************************************\nok: [localhost] => {"ansible_facts": {"default_playbook": {"playbook": "default"}}, "ansible_included_var_files": ["/opt/ansible/playbooks/default-playbook.yml"], "changed": false}\n\r\nTASK [Determine the version that is to be installed] ***************************\nok: [localhost] => {"ansible_facts": {"version": "default"}, "changed": false}\n\r\nTASK [Determine the default supported images for all known versions] ***********\nok: [localhost] => {"ansible_facts": {"supported_kiali_images": {"default": {"image_name": "quay.io/kiali/kiali", "image_version": "operator_version"}, "v1.12": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.12"}, "v1.24": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.24"}, "v1.36": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.36"}}}, "ansible_included_var_files": ["/opt/ansible/playbooks/default-supported-images.yml"], "changed": false}\n\r\nTASK [Override the supported images if found in the environment] ***************\nskipping: [localhost] => (item={'key': 'default', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'operator_version'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "default", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "operator_version"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.12', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.12'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.12", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.12"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.24', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.24'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.24", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.24"}}, "skip_reason": "Conditional result was False"}\nskipping: [localhost] => (item={'key': 'v1.36', 'value': {'image_name': 'quay.io/kiali/kiali', 'image_version': 'v1.36'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "v1.36", "value": {"image_name": "quay.io/kiali/kiali", "image_version": "v1.36"}}, "skip_reason": "Conditional result was False"}\n\r\nTASK [Examine environment and determine if supported image for the requested version is overridden even if no default is known] ***\nskipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}\n\r\nTASK [Make sure a default supported image or an override image is known] *******\nskipping: [localhost] => {"changed": false, "skip_reason": "Conditional result was False"}\n\r\nTASK [Run the version-specific deploy role] ************************************\n\r\nTASK [default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":[default/kiali-deploy : set_fact] *****************************************\nok: [localhost] => {"ansible_facts": {"k8s_plugin": "community.kubernetes.k8s"}, "changed": false}\n\r\nTASK [default/kiali-deploy : Get the original CR as-is for the camelCase keys and so we can update its status field] *\n: {"f:deployment": {".": {}, "f:instanceName": {}, "f:namespace": {}}, "f:environment": {".": {}, "f:isKubernetes": {}}, "f:progress": {}}}, "manager": "OpenAPI-Generator", "operation": "Update", "time": "2022-03-01T15:03:14Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:environment": {"f:kubernetesVersion": {}, "f:operatorVersion": {}}, "f:progress": {"f:duration": {}, "f:message": {}}}}, "manager": "OpenAPI-Generator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:02Z"}, {"apiVersion": "kiali.io/v1alpha1", "fieldsType": "FieldsV1", "fieldsV1": {"f:status": {"f:conditions": {}}}, "manager": "ansible-operator", "operation": "Update", "subresource": "status", "time": "2024-11-04T13:49:11Z"}], "name": "kiali", "namespace": "kiali-operator", "resourceVersion": "1040866186", "uid": "13e1d699-94f5-4545-a8c2-b8ce37d56c6f"}, "spec": {"auth": {"strategy": "anonymous"}, "deployment": {"accessible_namespaces": [""], "override_ingress_yaml": {"spec": {"rules": [{"host": "kiali.kiali-operator.ctn0.pic.services.pack", "http": {"paths": [{"backend": {"service": {"name": "kiali", "port": {"number": 20001}}}, "path": "/kiali", "pathType": "Prefix"}]}}]}}}, "external_services": {"prometheus": {"url":'

[ChrsMark]

Thank's @Masmiiadm! One more question: what container runtime your k8s cluster runs with?

Actually what we need to check here is what is stored in the log file at /var/log/pods/kiali-operator_kiali-kiali-operator-5f5f969b89-7vqcx_bd93551f-ff0e-413e-95b1-4fd8420f2690/operator/0.log (sorry for not mentioning that earlier). Could you tail that file and share a complete log line?

I think that for cri (containerd and cri-o) formats even with max_log_size set the parser should be able to detect the format since the format information is only on the "head" of the line. For Docker I don't think we can do much since it's actually json and if the json is incomplete the unmarshaling will fail anyways.