Fix Kafka Producer transactions shown as unknown on kibana apm dashboard

[heisGarvit]

Transactions with kakfa producers are shown as unknown on elastic apm kibana dashboard, while transaction with kafka consumer are shown as messaging. This pull request adds changes to fixes the issue.

Opentelemetry Instrumentation code for instrumenting producing message on kafka:

Producer Transactions visible as unknown on Kibana Dashboard:

Opentelemetry Instrumentation code for instrumenting consuming message from kafka:

Consumer Transactions visible as messaging on Kibana Dashboard:

After the Pull request changes:

[dmathieu]

Doing this would break our specifications: https://github.com/elastic/apm/blob/main/specs/agents/tracing-api-otel.md#transaction-type

The reason producer transactions are not defined is that transactions on the elastic model are root spans.
Messages should never be produced as a root span. They would be meaningless (you know something was produced, but you wan't know what did it).

An incoming HTTP request, or a message being received would be root spans (to a service), because those are entrypoints into said service.

Within spans, we do mark producer ones as messaging when the messaging.system attribute is set:

apm-agent-go/module/apmotel/span.go

Lines 261 to 262 in 2168456

	case "messaging.system":
	s.span.Type = "messaging"

So while I wouldn't be opposed to applying this change (it would need tests), the fact that you need it might be a smell within your own flow.

[heisGarvit]

@dmathieu

The reason producer transactions are not defined is that transactions on the elastic model are root spans.
Messages should never be produced as a root span. They would be meaningless (you know something was produced, but you wan't know what did it).

Root span should not be producer, makes sense, thanks for pointing out.