[APM] "Trace/Logs" screen not only displays log messages that have the trace.id fields matching but also log messages that contain the traceId value in their message #126342
Description
Kibana version:
8.0.0
Elasticsearch version:
8.0.0
Original install method (e.g. download page, yum, from source, etc.):
Elastic Cloud
Describe the bug:
The APM Trace/Logs not only displays log messages that have the trace.id fields matching but also all log message that contain the traceId value in their message
Steps to reproduce:
- Instrument an application with Elastic APM or OpenTelemetry tracing
- capture the access logs of the Kibana server used to visualize the health of this application (Kibana uses the traceId as a URL parameter)
- Look at the Trace view logs in Elastic APM, it will mix the log messageds of the application with the log messages of Kibana APM related to this application
Expected behavior:
Only the logs of the application trace are rendered.
Screenshots (if relevant):
Mix of log messages for the trace 1629211caf1f494f6e888a35485f38c5
| Field | Value |
|---|---|
| @timestamp | 2022-02-24T11:35:07.025Z |
| agent.ephemeral_id | 728b261b-c410-4e78-9d0d-29bedc4f42f7 |
| agent.id | 9982bd2e-a325-4e28-9a60-d49df890adb5 |
| agent.name | gke-otel-oblt-otel-oblt-pool-1d91b874-02vr |
| agent.type | filebeat |
| agent.version | 8.2.0 |
| cloud.account.id | elastic-observability |
| cloud.availability_zone | us-central1-c |
| cloud.instance.id | 3096344924463540872 |
| cloud.instance.name | gke-otel-oblt-otel-oblt-pool-1d91b874-02vr |
| cloud.machine.type | n1-standard-4 |
| cloud.project.id | elastic-observability |
| cloud.provider | gcp |
| cloud.service.name | GCE |
| container.id | d4acdd5790d449c7a1db187bb702ed965d03a951e7e0e0f6d6efa997a5c6204c |
| container.image.name | k8s.gcr.io/ingress-nginx/controller:v1.1.0@sha256:f766669fdcf3dc26347ed273a55e754b427eb4411ee075a53f30718b4499076a |
| container.runtime | containerd |
| data_stream.dataset | generic |
| data_stream.namespace | default |
| data_stream.type | logs |
| ecs.version | 8.0.0 |
| elastic_agent.id | 9982bd2e-a325-4e28-9a60-d49df890adb5 |
| elastic_agent.snapshot | TRUE |
| elastic_agent.version | 8.2.0 |
| event.dataset | generic |
| host.architecture | x86_64 |
| host.containerized | TRUE |
| host.hostname | gke-otel-oblt-otel-oblt-pool-1d91b874-02vr |
| host.ip | "10.128.0.47 |
| fe80::4001:aff:fe80:2f | |
| 169.254.123.1 | |
| 10.88.0.1 | |
| fe80::3ce1:79ff:fe24:49cc | |
| 10.88.0.1 | |
| fe80::8468:72ff:feb9:ba43 | |
| 10.88.0.1 | |
| fe80::9450:b2ff:fe15:bec1 | |
| 10.88.0.1 | |
| fe80::b873:5bff:fecd:6d54 | |
| 10.88.0.1 | |
| fe80::d8ef:9bff:fe53:f09a | |
| 10.88.0.1 | |
| fe80::e06d:afff:fed4:5944 | |
| 10.88.0.1 | |
| fe80::c0d:29ff:fe7d:b2e1 | |
| 10.88.0.1 | |
| fe80::d86b:51ff:fee9:c1be | |
| 10.88.0.1 | |
| fe80::bcf7:62ff:fef2:7925 | |
| 10.88.0.1 | |
| fe80::6476:f7ff:fe95:5bdb | |
| 10.88.0.1 | |
| fe80::2c27:afff:fed2:7212 | |
| 10.88.0.1 | |
| fe80::5073:6eff:fe06:e785" | |
| host.mac | "42:01:0a:80:00:2f |
| 02:42:b5:ed:99:7e | |
| 3e:e1:79:24:49:cc | |
| 86:68:72:b9:ba:43 | |
| 96:50:b2:15:be:c1 | |
| ba:73:5b:cd:6d:54 | |
| da:ef:9b:53:f0:9a | |
| e2:6d:af:d4:59:44 | |
| 0e:0d:29:7d:b2:e1 | |
| da:6b:51:e9:c1:be | |
| be:f7:62:f2:79:25 | |
| 66:76:f7:95:5b:db | |
| 2e:27:af:d2:72:12 | |
| 52:73:6e:06:e7:85" | |
| host.name | gke-otel-oblt-otel-oblt-pool-1d91b874-02vr |
| host.os.codename | focal |
| host.os.family | debian |
| host.os.kernel | 5.4.144+ |
| host.os.name | Ubuntu |
| host.os.platform | ubuntu |
| host.os.type | linux |
| host.os.version | 20.04.3 LTS (Focal Fossa) |
| input.type | log |
| kubernetes.container.name | controller |
| kubernetes.deployment.name | lb-ingress-nginx-controller |
| kubernetes.labels.app_kubernetes_io/component | controller |
| kubernetes.labels.app_kubernetes_io/instance | lb |
| kubernetes.labels.app_kubernetes_io/name | ingress-nginx |
| kubernetes.labels.pod-template-hash | 65bdcc8c88 |
| kubernetes.namespace | ingress-nginx |
| kubernetes.namespace_labels.k8s-app | ingress-nginx |
| kubernetes.namespace_labels.kubernetes_io/metadata_name | ingress-nginx |
| kubernetes.namespace_uid | 9174cc51-2154-4a3f-a1e9-2656123a8b75 |
| kubernetes.node.hostname | gke-otel-oblt-otel-oblt-pool-1d91b874-02vr.c.elastic-observability.internal |
| kubernetes.node.labels.beta_kubernetes_io/arch | amd64 |
| kubernetes.node.labels.beta_kubernetes_io/instance-type | n1-standard-4 |
| kubernetes.node.labels.beta_kubernetes_io/os | linux |
| kubernetes.node.labels.cloud_google_com/gke-boot-disk | pd-standard |
| kubernetes.node.labels.cloud_google_com/gke-container-runtime | containerd |
| kubernetes.node.labels.cloud_google_com/gke-nodepool | otel-oblt-pool |
| kubernetes.node.labels.cloud_google_com/gke-os-distribution | cos |
| kubernetes.node.labels.cloud_google_com/machine-family | n1 |
| kubernetes.node.labels.failure-domain_beta_kubernetes_io/region | us-central1 |
| kubernetes.node.labels.failure-domain_beta_kubernetes_io/zone | us-central1-c |
| kubernetes.node.labels.kubernetes_io/arch | amd64 |
| kubernetes.node.labels.kubernetes_io/hostname | gke-otel-oblt-otel-oblt-pool-1d91b874-02vr |
| kubernetes.node.labels.kubernetes_io/os | linux |
| kubernetes.node.labels.node_kubernetes_io/instance-type | n1-standard-4 |
| kubernetes.node.labels.topology_gke_io/zone | us-central1-c |
| kubernetes.node.labels.topology_kubernetes_io/region | us-central1 |
| kubernetes.node.labels.topology_kubernetes_io/zone | us-central1-c |
| kubernetes.node.name | gke-otel-oblt-otel-oblt-pool-1d91b874-02vr |
| kubernetes.node.uid | 0d895ab8-ecfe-4f8d-b05c-c1db19301830 |
| kubernetes.pod.ip | 10.88.0.11 |
| kubernetes.pod.name | lb-ingress-nginx-controller-65bdcc8c88-2sx64 |
| kubernetes.pod.uid | dde1be57-7514-418d-8e17-e5c8ec925ae5 |
| kubernetes.replicaset.name | lb-ingress-nginx-controller-65bdcc8c88 |
| log.file.path | /var/log/containers/lb-ingress-nginx-controller-65bdcc8c88-2sx64_ingress-nginx_controller-d4acdd5790d449c7a1db187bb702ed965d03a951e7e0e0f6d6efa997a5c6204c.log |
| log.offset | 1602758 |
| message | 2022-02-24T11:35:05.477874625Z stdout F 185.141.240.115 - - [24/Feb/2022:11:35:05 +0000] "GET /job/hello/job/helloWorld-k8s/2/ecfdb20f4f3c4746a858d667ad027c97.us-west2.gcp.elastic-cloud.com/app/ HTTP/1.1" 404 199 "http://jenkins.34.133.113.14.ip.es.io/job/hello/job/helloWorld-k8s/2/ecfdb20f4f3c4746a858d667ad027c97.us-west2.gcp.elastic-cloud.com/app/logs/stream?logPosition=(end:now,start:now-40d,streamLive:!f)&logFilter=(language:kuery,query:%27trace.id:1629211caf1f494f6e888a35485f38c5%27)&" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36" 1241 0.002 [default-jenkins-8080] [] 10.88.2.149:8080 199 0.002 404 b001aaf9c07df5cfcc04fa90a9e059cf |
| orchestrator.cluster.name | otel-oblt |
| orchestrator.cluster.url | https://34.71.150.17 |
Errors in browser console (if relevant):
Provide logs and/or server output (if relevant):
Any additional context: