Omit Null values from Structured Metadata

[ZIRAKrezovic]

Thanks for finally adding Structured Metadata support. I would like to propose an enhancement

Could you add either an option to disable, or disable by default, sending of fields with null/empty value?

I would like to use

<structuredMetadataPattern>trace_id=%mdc{traceId}</structuredMetadataPattern>

But when no traceId is present in mdc, in Loki I see the following

This breaks navigation from traces to logs, as empty link is generated

https://grafana.com/docs/grafana/latest/datasources/tempo/link-trace-id/

resulting in an empty screen as trace id is not populated

Using none shown in example results in same problem - none is not a valid trace id in tempo, at any time.

See discussion for logstash-logback-encoder and same feature request for ideas logfellow/logstash-logback-encoder#201

[nehaev]

Hi @ZIRAKrezovic, thanks for reporting this! I agree with you, this functionality should be implemented in Loki4j.

As a workaround, I suggest to set 2 different Lok4j appenders: one with MDC property and another one without it. Then you can route log events between them using filters.

[nehaev]

This was previously raised in #207, but closed as it was impossible to implement back then. Now that #265 is merged, this issue needs some re-evaluation.

[ZIRAKrezovic]

I use otel-collector as a receiver, so I can transform it there into OTLP compatible payload

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318
  loki:
    protocols:
      http:
        endpoint: 0.0.0.0:3100
    use_incoming_timestamp: true

processors:
  batch:
  memory_limiter:
    check_interval: 5s
    limit_mib: 256
    spike_limit_mib: 128
  transform/loki:
    log_statements:
    - context: log
      statements:
      - set(resource.attributes["service.name"], attributes["application"])
      - set(resource.attributes["application"], attributes["application"])
      - set(resource.attributes["hostname"], attributes["hostname"])
      - set(resource.attributes["vendor"], attributes["vendor"])
    - context: log
      conditions:
      - IsMap(attributes) and attributes["level"] != nil and Len(attributes["level"]) > 0
      statements:
      - set(severity_text, attributes["level"])
    - context: log
      conditions:
      - IsMap(attributes) and attributes["trace_id"] != nil and Len(attributes["trace_id"]) > 0
      statements:
      - set(trace_id.string, attributes["trace_id"])
    - context: log
      conditions:
      - IsMap(attributes) and attributes["span_id"] != nil and Len(attributes["span_id"]) > 0
      statements:
      - set(span_id.string, attributes["span_id"])
    - context: log
      statements:
      - delete_key(attributes, "application")
      - delete_key(attributes, "hostname")
      - delete_key(attributes, "vendor")
      - delete_key(attributes, "trace_id")
      - delete_key(attributes, "span_id")

exporters:
  debug:
    verbosity: detailed
  otlphttp/loki:
    endpoint: redacted

service:
  pipelines:
    logs:
      receivers: [otlp, loki]
      processors: [transform/loki, batch, memory_limiter]
      exporters: [otlphttp/loki, debug]