MeshAccessLogs: Multiple lines of logs are consumed as single line by Dynatrace

[lukidzi]

What happened?

Dynatrace requires [timestamp] [logLevel] {json_data} to correlate data as a single log. Because JSON format produced by Envoy is a full json string without these prefixes, logs are not properly indexed by Dynatrace. It would be nice to support this format or HTTP backend

type: MeshAccessLog
name: accesslogs
mesh: default
spec:
 targetRef:
   kind: Mesh
 from:
   - targetRef:
       kind: Mesh
     default:
       backends:
         - type: File
           file:
             path: /dev/stdout
             format:
               type: Json
                 json:
                   - key: "start_time"
                      value: "%START_TIME%"
                   - key: "bytes_received"
                      value: "%BYTES_RECEIVED%"

[bartsmykla]

triage: it's possible that Dynatrace support OTEL gRPC endpoints, which should solve this issue. We need to check if that's possible

[Icarus9913]

I found the way to handle with the stdout in json type.
Use the MeshProxyPatch to supplement the JsonFormatOptions sort_properties (The output JSON string properties will be sorted.)
In this way, each log would be single one that was analyzed by Dynatrace

apiVersion: kuma.io/v1alpha1
kind: MeshAccessLog
metadata:
  name: default
  namespace: kuma-system
  labels:
    kuma.io/mesh: default
spec:
  targetRef:
    kind: Mesh
  from:
    - targetRef:
        kind: Mesh
      default:
        backends:
          - type: File
            file:
              path: /dev/stdout
              format:
                type: Json
                json:
                - key: "data_set"
                  value: "kuma"
                - key: "start_time"
                  value: "%START_TIME(%Y-%m-%d %H:%M:%S %s)%"
                - key: "kuma_mesh"
                  value: "%KUMA_MESH%"
                - key: "kuma_source_address_without_port"
                  value: "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%"
                - key: "kuma_source_service"
                  value: "%KUMA_SOURCE_SERVICE%"
                - key: "upstream_host"
                  value: "%UPSTREAM_HOST%"
                - key: "kuma_destination_service"
                  value: "%KUMA_DESTINATION_SERVICE%"
                - key: "duration_ms"
                  value: "%DURATION%"
                - key: "bytes_sent"
                  value: "%BYTES_SENT%"
                - key: "bytes_received"
                  value: "%BYTES_RECEIVED%" 
---
apiVersion: kuma.io/v1alpha1
kind: MeshProxyPatch
metadata:
  name: accesslog
  namespace: kuma-system
spec:
  targetRef:
    kind: Mesh
  default:
    appendModifications:
      - listener:
          operation: Patch
          match:
            origin: inbound
            tags:
              k8s.kuma.io/service-name: httpbin-1
          jsonPatches:
            - op: add
              path: /filterChains/0/filters/0/typedConfig/accessLog/0/typedConfig/logFormat/jsonFormatOptions
              value: {"sort_properties": true}

[Icarus9913]

Maybe we should also add that jsonFormatOptions property in our MeshAccessLog policy configuration

kuma/pkg/plugins/policies/meshaccesslog/plugin/xds/configurer.go

Lines 163 to 172 in e21c751

	func (c *Configurer) sfsJSON(fields map[string]*structpb.Value, omitEmpty bool) *envoy_core.SubstitutionFormatString {
	return &envoy_core.SubstitutionFormatString{
	Format: &envoy_core.SubstitutionFormatString_JsonFormat{
	JsonFormat: &structpb.Struct{
	Fields: fields,
	},
	},
	OmitEmptyValues: omitEmpty,
	}
	}

[michaelbeaumont]

Waiting on feedback from user on MeshProxyPatch

[slonka]

Still waiting, pinged on slack.

[jakubdyszkiewicz]

@Icarus9913 did you try this with Dynatrace?

[Icarus9913]

Yeah, I did a test from my side with Dynatrace and it works in my case.

---

However, it seems that not works for the user. Maybe we should suggest the user to switch to OTEL way and get a try

[lobkovilya]

Triage: seems like it's not an issue. Closing the issue.