Allow jaeger supported connections to service graph

[trajano]

Component(s)

connector/servicegraph

Is your feature request related to a problem? Please describe.

I used to use Jaeger to generate a service graph for the most part I didn't have to do anything special, but when I am using the servicegraph component it appears to be restricted as denoted here

opentelemetry-collector-contrib/connector/servicegraphconnector/README.md

Lines 47 to 51 in 1d4cf91

	It currently supports the following requests:
	* A direct request between two services where the outgoing and the incoming span must have `span.kind` client and server respectively.
	* A request across a messaging system where the outgoing and the incoming span must have `span.kind` producer and consumer respectively.
	* A database request; in this case the connector looks for spans containing attributes `span.kind`=client as well as db.name.

Describe the solution you'd like

Just make it generate similar graphs to Jaeger.

I think a good one would be to simply check whether a trace contains a span that goes from one service.name to another. This is one that Jaeger would support

Here's another one with Spring Boot connecting to Redis where net.sock.peer.name can be used as a fallback to db.name to specify a remote non-tracked service (note Jaeger doesn't support this one either)

Describe alternatives you've considered

Go back to Jaeger

Using something like this (but that didn't work)

  transform/set_span_kind_client:
    trace_statements:
      - context: span
        statements:
          - set(attributes["kind"], "client") where attributes["kind"] == nil

Additional context

I do get data specifically from Grafana services

# HELP traces_service_graph_request_total 
# TYPE traces_service_graph_request_total counter
traces_service_graph_request_total{client="grafana",connection_type="",failed="false",server="prometheus"} 540

So I know it's working.

[github-actions]

Pinging code owners:

• connector/servicegraph: @jpkrohling @mapno

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[trajano]

I think an easy way is to remove the requirement that the source is a client so you'd allow a connection when it is server to server. I am presuming the change would be around here

opentelemetry-collector-contrib/processor/servicegraphprocessor/processor.go

Line 226 in 1d4cf91

func (p *serviceGraphProcessor) aggregateMetrics(ctx context.Context, td ptrace.Traces) (err error) {

Maybe something like

switch span.Kind() {
  case ptrace.SpanKindProducer:
    ...
    fallthrough
  case ptrace.SpanKindConsumer: // this is shifted up
    ...
    fallthrough
  case ptrace.SpanKindServer: // this is shifted up
    ...
    fallthrough // fall through since server can be client as well.
  case ptrace.SpanKindClient:
    ...

[trajano]

For the connection using net.sock.peer.name it can be done using

  transform/set_db_name
    trace_statements:
      - context: span
        statements:
          - set(attributes["db.name"], attributes["net.sock.peer.name"]) where attributes["db.name"] == nil and attributes["db.system"] != nil

[mapno]

Hi! Not using kind to identify requests is something we've considered in the past, but haven't implemented. We've preferred adhering to OTel semantic conventions, which define that spans must be tagged with kind when they're part of a request (ref).

I'm not opposed to a different approach, but there are tradeoffs that need to be considered first: not using kind is way more resource intensive than doing so, as any span can be a client or a server—which results in a lot more data buffered and more calculations. If you have a proposal in mind, I'm happy to review it.

Finally, theres is a feature called virtual nodes that could help you with some of the cases you mention. The docs linked are from Grafana Tempo, but it's the same implementation.

[crobert-1]

@trajano Have you had a chance to check out virtual nodes as @mapno referenced? Do you have other thoughts on @mapno's response here?

[trajano]

Looks fair but I stopped using Jaeger for tracing now and switched to Tempo so I am closing this. Maybe someone who is still using Jaeger can try it out.

In addition traefik/traefik#10223 may solve the issues as well since this fixes a lot of the OTel stuff.