OpenTelemetry plugin: non-compliant HTTP client spans

[carsonip]

Is there an existing issue for this?

• I have searched the existing issues

Kong version ($ kong version)

kong/kong-gateway:3.9.0.0

Current Behavior

Kong OpenTelemetry plugin creates HTTP client spans that do not follow SemConv HTTP spans spec in https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client . Many required attributes are missing, causing issues in observability backends like Elastic. Elastic is planning to document this as a known issue with Kong OpenTelemetry plugin in elastic/apm-server#16446.

To give an example, consider a set of spans emitted by Kong that describes a http call from service B to C via Kong. While the top level span (id 052dc3538172140c) contains http attributes as described in Kong docs, the client span (id 3b8c984527a7f95e) from Kong to service C contains only net.peer.name, net.peer.port and net.peer.ip which is not spec-compliant as a HTTP client span. This is insufficient for observability backends to show this as a call from Kong to service C.

OTel collector logs from debug exporter:

opentelemetry-collector-1  | 2025-03-27T10:03:10.033Z   info    ResourceSpans #0
opentelemetry-collector-1  | Resource SchemaURL: 
opentelemetry-collector-1  | Resource attributes:
opentelemetry-collector-1  |      -> service.name: Str(kong-dev)
opentelemetry-collector-1  |      -> service.instance.id: Str(f8636234-6a5d-4420-86df-4d89ec1c35a4)
opentelemetry-collector-1  |      -> service.version: Str(3.9.0.0)
opentelemetry-collector-1  | ScopeSpans #0
opentelemetry-collector-1  | ScopeSpans SchemaURL: 
opentelemetry-collector-1  | InstrumentationScope kong-internal 0.1.0
opentelemetry-collector-1  | Span #0
opentelemetry-collector-1  |     Trace ID       : bed5e28cfe42774e8ef1c9509685001d
opentelemetry-collector-1  |     Parent ID      : 5c80856b11ebecf5
opentelemetry-collector-1  |     ID             : 052dc3538172140c
opentelemetry-collector-1  |     Name           : kong
opentelemetry-collector-1  |     Kind           : Server
opentelemetry-collector-1  |     Start time     : 2025-03-27 10:03:09.02 +0000 UTC
opentelemetry-collector-1  |     End time       : 2025-03-27 10:03:09.030311168 +0000 UTC
opentelemetry-collector-1  |     Status code    : Unset
opentelemetry-collector-1  |     Status message : 
opentelemetry-collector-1  | Attributes:
opentelemetry-collector-1  |      -> http.route: Str(/c)
opentelemetry-collector-1  |      -> http.host: Str(172.17.0.1)
opentelemetry-collector-1  |      -> http.status_code: Int(200)
opentelemetry-collector-1  |      -> http.flavor: Str(1.1)
opentelemetry-collector-1  |      -> http.method: Str(GET)
opentelemetry-collector-1  |      -> http.url: Str(http://172.17.0.1/c)
opentelemetry-collector-1  |      -> http.client_ip: Str(172.23.0.1)
opentelemetry-collector-1  |      -> kong.request.id: Str(0ceff764b59cd74d7afcfb78108c435a)
opentelemetry-collector-1  |      -> http.scheme: Str(http)
opentelemetry-collector-1  |      -> net.peer.ip: Str(172.23.0.1)
opentelemetry-collector-1  | Span #1
opentelemetry-collector-1  |     Trace ID       : bed5e28cfe42774e8ef1c9509685001d
opentelemetry-collector-1  |     Parent ID      : 052dc3538172140c
opentelemetry-collector-1  |     ID             : ddac0dddf603c464
opentelemetry-collector-1  |     Name           : kong.router
opentelemetry-collector-1  |     Kind           : Internal
opentelemetry-collector-1  |     Start time     : 2025-03-27 10:03:09.021509888 +0000 UTC
opentelemetry-collector-1  |     End time       : 2025-03-27 10:03:09.023940096 +0000 UTC
opentelemetry-collector-1  |     Status code    : Unset
opentelemetry-collector-1  |     Status message : 
opentelemetry-collector-1  | Span #2
opentelemetry-collector-1  |     Trace ID       : bed5e28cfe42774e8ef1c9509685001d
opentelemetry-collector-1  |     Parent ID      : 052dc3538172140c
opentelemetry-collector-1  |     ID             : 8d1519601bd32c6b
opentelemetry-collector-1  |     Name           : kong.access.plugin.opentelemetry
opentelemetry-collector-1  |     Kind           : Internal
opentelemetry-collector-1  |     Start time     : 2025-03-27 10:03:09.025487104 +0000 UTC
opentelemetry-collector-1  |     End time       : 2025-03-27 10:03:09.027902464 +0000 UTC
opentelemetry-collector-1  |     Status code    : Unset
opentelemetry-collector-1  |     Status message : 
opentelemetry-collector-1  | Span #3
opentelemetry-collector-1  |     Trace ID       : bed5e28cfe42774e8ef1c9509685001d
opentelemetry-collector-1  |     Parent ID      : 052dc3538172140c
opentelemetry-collector-1  |     ID             : 3e127115c3edbce0
opentelemetry-collector-1  |     Name           : kong.header_filter.plugin.opentelemetry
opentelemetry-collector-1  |     Kind           : Internal
opentelemetry-collector-1  |     Start time     : 2025-03-27 10:03:09.030148608 +0000 UTC
opentelemetry-collector-1  |     End time       : 2025-03-27 10:03:09.030160128 +0000 UTC
opentelemetry-collector-1  |     Status code    : Unset
opentelemetry-collector-1  |     Status message : 
opentelemetry-collector-1  | Span #4
opentelemetry-collector-1  |     Trace ID       : bed5e28cfe42774e8ef1c9509685001d
opentelemetry-collector-1  |     Parent ID      : 052dc3538172140c
opentelemetry-collector-1  |     ID             : 3b8c984527a7f95e
opentelemetry-collector-1  |     Name           : kong.balancer
opentelemetry-collector-1  |     Kind           : Client
opentelemetry-collector-1  |     Start time     : 2025-03-27 10:03:09.028379392 +0000 UTC
opentelemetry-collector-1  |     End time       : 2025-03-27 10:03:09.030311168 +0000 UTC
opentelemetry-collector-1  |     Status code    : Unset
opentelemetry-collector-1  |     Status message : 
opentelemetry-collector-1  | Attributes:
opentelemetry-collector-1  |      -> try_count: Double(1)
opentelemetry-collector-1  |      -> net.peer.name: Str(172.17.0.1)
opentelemetry-collector-1  |      -> net.peer.port: Double(10082)
opentelemetry-collector-1  |      -> net.peer.ip: Str(172.17.0.1)

Expected Behavior

Kong OpenTelemetry Plugin should ensure client spans from kong.balancer follow spec https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client

Steps To Reproduce

A minimally reproducible example is described in elastic/apm-server#16446

Anything else?

No response

[hoanbc]

Hi @carsonip do you know how to show detail transactions as path of request? , I did follow workaround but it not working, it still show "kong" for all transactions.
Link: https://support.konghq.com/support/s/article/OpenTelemetry-plugin-in-Kong-is-naming-all-transactions-statically-as-kong

[carsonip]

@hoanbc sorry for the delayed response. It is not clear whether it is related to this issue. Do you mind starting a thread in https://discuss.elastic.co/c/observability/82 with Elastic Observability, APM, with tag server to ensure timely response and avoid adding noise to this issue?

[hoanbc]

Hi @carsonip i has workaround with otel config as below. It will extract path after url.
processors:
batch: {}
attributes:
actions:
- key: http.url
action: extract
pattern: '^(?P<short_url>https?://[^/]+)(?P<url_truncated_path>/.*)?'
transform:
trace_statements:
- context: span
statements:
- set(name, Concat([attributes["http.method"], attributes["url_truncated_path"]], " ")) where name == "kong"

[carsonip]

Great to know, thanks for posting a workaround for your issue!