Rogue Parent ID generate in Azure Container App

[dickiebowuk]

Component(s)

No response

Describe the issue you're reporting

I have written a simple API based on the scaffolded weather app API.

Scenario 1: I have the API running on my laptop, using Swagger Docs to call the end points that calls an Azure SQL Database to get the weather forecasts and exporting the telemetry data to an OTel Collector Contrib running in an Azure Container App with the ingress set to accept traffic from anywhere. The span collected does not have a Parent ID for the InstrumentationScope Microsoft.AspNetCore section.

Scenario 2: I have the same API running in an Azure Container App within a secure VNet. I use an APIM to call the API endpoints and it uses the same Azure SQL Database to get the weather reports. It exports the Otel data to a collector in an Azure Container App running in the same VNet. The span collected in this environment has a rogue Parent ID in the InstrumentationScope Microsoft.AspNetCore section that is not included in the Otel data.

What could be causing Otel libraries to think there is a parent span that it is not including in the data?

OTel Console output for Scenario 1:

2023-11-02T15:56:55.649302952Z 2023-11-02T15:56:55.649Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 2}
2023-11-02T15:56:55.650107441Z 2023-11-02T15:56:55.649Z	info	ResourceSpans #0
2023-11-02T15:56:55.650140021Z Resource SchemaURL: 
2023-11-02T15:56:55.650149810Z Resource attributes:
2023-11-02T15:56:55.650158075Z      -> telemetry.sdk.name: Str(opentelemetry)
2023-11-02T15:56:55.650166351Z      -> telemetry.sdk.language: Str(dotnet)
2023-11-02T15:56:55.650174285Z      -> telemetry.sdk.version: Str(1.6.0)
2023-11-02T15:56:55.650182391Z      -> environment: Str(Development)
2023-11-02T15:56:55.650190346Z      -> service.name: Str(Otelapiexample.API.Development)
2023-11-02T15:56:55.650210353Z      -> service.instance.id: Str(a2dfbcf1-4342-4202-aa86-d858a331fd80)
2023-11-02T15:56:55.650218919Z ScopeSpans #0
2023-11-02T15:56:55.650227275Z ScopeSpans SchemaURL: 
2023-11-02T15:56:55.650235820Z InstrumentationScope OpenTelemetry.Instrumentation.SqlClient 1.0.0.0
2023-11-02T15:56:55.650244396Z Span #0
2023-11-02T15:56:55.650252772Z     Trace ID       : e79bfa6236b05ace3a85656f5abe71c2
2023-11-02T15:56:55.650261508Z     Parent ID      : 1dcf7d47cef0ed68
2023-11-02T15:56:55.650270145Z     ID             : dc607b3c2ca2ced4
2023-11-02T15:56:55.650278680Z     Name           : WeatherForecast
2023-11-02T15:56:55.650287337Z     Kind           : Client
2023-11-02T15:56:55.650295332Z     Start time     : 2023-11-02 15:56:41.944259 +0000 UTC
2023-11-02T15:56:55.650303938Z     End time       : 2023-11-02 15:56:41.9706715 +0000 UTC
2023-11-02T15:56:55.650312123Z     Status code    : Unset
2023-11-02T15:56:55.650320328Z     Status message : 
2023-11-02T15:56:55.650341909Z Attributes:
2023-11-02T15:56:55.650356626Z      -> db.system: Str(mssql)
2023-11-02T15:56:55.650365212Z      -> db.name: Str(WeatherForecast)
2023-11-02T15:56:55.650373328Z      -> net.peer.name: Str(appdrabtest.database.windows.net)
2023-11-02T15:56:55.650382505Z      -> db.statement_type: Str(StoredProcedure)
2023-11-02T15:56:55.650391401Z      -> db.statement: Str(prWeatherSummaryList)
2023-11-02T15:56:55.650400839Z      -> sqlCommand.commandTimeOut: Str(30)
2023-11-02T15:56:55.650410286Z      -> peer.service: Str(appdrabtest.database.windows.net)
2023-11-02T15:56:55.650429633Z      -> collector.hostname: Str(collector--6eit1ja-544d484b97-8pnxn)
2023-11-02T15:56:55.650492510Z      -> azure.container_app.revision: Str(collector--6eit1ja)
2023-11-02T15:56:55.650584001Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:56:55.650616963Z      -> source.blog: Str(true)
2023-11-02T15:56:55.650645125Z ScopeSpans #1
2023-11-02T15:56:55.650978658Z ScopeSpans SchemaURL: 
2023-11-02T15:56:55.650999678Z InstrumentationScope Microsoft.AspNetCore 
2023-11-02T15:56:55.651022490Z Span #0
2023-11-02T15:56:55.651202797Z     Trace ID       : e79bfa6236b05ace3a85656f5abe71c2
2023-11-02T15:56:55.651221091Z     Parent ID      : 
2023-11-02T15:56:55.651261667Z     ID             : 1dcf7d47cef0ed68
2023-11-02T15:56:55.651282877Z     Name           : api/WeatherForecast/list
2023-11-02T15:56:55.651292695Z     Kind           : Server
2023-11-02T15:56:55.651301622Z     Start time     : 2023-11-02 15:56:41.9231418 +0000 UTC
2023-11-02T15:56:55.651310288Z     End time       : 2023-11-02 15:56:41.9843922 +0000 UTC
2023-11-02T15:56:55.651318944Z     Status code    : Unset
2023-11-02T15:56:55.651327780Z     Status message : 
2023-11-02T15:56:55.651335916Z Attributes:
2023-11-02T15:56:55.651344322Z      -> net.host.name: Str(localhost)
2023-11-02T15:56:55.651353128Z      -> net.host.port: Int(7200)
2023-11-02T15:56:55.651358989Z      -> http.method: Str(GET)
2023-11-02T15:56:55.651364640Z      -> http.scheme: Str(https)
2023-11-02T15:56:55.651370310Z      -> http.target: Str(/api/WeatherForecast/list)
2023-11-02T15:56:55.651376031Z      -> http.url: Str(https://localhost:7200/api/WeatherForecast/list)
2023-11-02T15:56:55.651381711Z      -> http.flavor: Str(2.0)
2023-11-02T15:56:55.651388614Z      -> http.user_agent: Str(Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36)
2023-11-02T15:56:55.651397811Z      -> requestProtocol: Str(HTTP/2)
2023-11-02T15:56:55.651406778Z      -> http.route: Str(api/WeatherForecast/list)
2023-11-02T15:56:55.651414853Z      -> http.status_code: Int(200)
2023-11-02T15:56:55.651423129Z      -> collector.hostname: Str(collector--)
2023-11-02T15:56:55.651431364Z      -> azure.container_app.revision: Str(collector--)
2023-11-02T15:56:55.651437125Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:56:55.651442705Z      -> source.blog: Str(true)
2023-11-02T15:56:55.651448346Z 	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2023-11-02T15:56:55.651469055Z 2023-11-02T15:56:55.649Z	debug	otlphttpexporter@v0.67.0/otlp.go:127	Preparing to make HTTP request	{"kind": "exporter", "data_type": "traces", "name": "otlphttp/withauth", "url": "https://url/data/v1/trace"}

Otel log for Scenario 2:

2023-11-02T15:57:49.923527097Z 2023-11-02T15:57:49.923Z	info	TracesExporter	{"kind": "exporter", "data_type": "traces", "name": "logging", "#spans": 2}
2023-11-02T15:57:49.924143138Z 2023-11-02T15:57:49.923Z	info	ResourceSpans #0
2023-11-02T15:57:49.924161152Z Resource SchemaURL: 
2023-11-02T15:57:49.924170489Z Resource attributes:
2023-11-02T15:57:49.924189435Z      -> telemetry.sdk.name: Str(opentelemetry)
2023-11-02T15:57:49.924197370Z      -> telemetry.sdk.language: Str(dotnet)
2023-11-02T15:57:49.924600348Z      -> telemetry.sdk.version: Str(1.6.0)
2023-11-02T15:57:49.924758661Z      -> environment: Str(Development)
2023-11-02T15:57:49.924807142Z      -> service.name: Str(Otelapiexample.API.Development.Azure)
2023-11-02T15:57:49.924818754Z      -> service.instance.id: Str(819c8322-2d73-448e-ae9c-4adebdc878f2)
2023-11-02T15:57:49.924827450Z ScopeSpans #0
2023-11-02T15:57:49.924835886Z ScopeSpans SchemaURL: 
2023-11-02T15:57:49.924844352Z InstrumentationScope OpenTelemetry.Instrumentation.SqlClient 1.0.0.0
2023-11-02T15:57:49.924856725Z Span #0
2023-11-02T15:57:49.924866213Z     Trace ID       : ea9b45ad8197256c0792e276e3e7cd4f
2023-11-02T15:57:49.924886331Z     Parent ID      : 6fe8f6d99948db15
2023-11-02T15:57:49.924894937Z     ID             : d4cecc62bc5a5684
2023-11-02T15:57:49.924903603Z     Name           : WeatherForecast
2023-11-02T15:57:49.924914604Z     Kind           : Client
2023-11-02T15:57:49.924922338Z     Start time     : 2023-11-02 15:57:40.6872252 +0000 UTC
2023-11-02T15:57:49.924954539Z     End time       : 2023-11-02 15:57:40.8726132 +0000 UTC
2023-11-02T15:57:49.924964487Z     Status code    : Unset
2023-11-02T15:57:49.924972903Z     Status message : 
2023-11-02T15:57:49.924984234Z Attributes:
2023-11-02T15:57:49.925398904Z      -> db.system: Str(mssql)
2023-11-02T15:57:49.925428810Z      -> db.name: Str(WeatherForecast)
2023-11-02T15:57:49.925437597Z      -> net.peer.name: Str(test.database.windows.net)
2023-11-02T15:57:49.925445582Z      -> db.statement_type: Str(StoredProcedure)
2023-11-02T15:57:49.925453196Z      -> db.statement: Str(prWeatherSummaryList)
2023-11-02T15:57:49.925460660Z      -> sqlCommand.commandTimeOut: Str(30)
2023-11-02T15:57:49.925468364Z      -> peer.service: Str(appdrabtest.database.windows.net)
2023-11-02T15:57:49.925478443Z      -> collector.hostname: Str(collector--)
2023-11-02T15:57:49.925486929Z      -> azure.container_app.revision: Str(collector--)
2023-11-02T15:57:49.925495005Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:57:49.925502699Z      -> source.blog: Str(true)
2023-11-02T15:57:49.925510614Z ScopeSpans #1
2023-11-02T15:57:49.925518238Z ScopeSpans SchemaURL: 
2023-11-02T15:57:49.925526193Z InstrumentationScope Microsoft.AspNetCore 
2023-11-02T15:57:49.925534108Z Span #0
2023-11-02T15:57:49.925542363Z     Trace ID       : ea9b45ad8197256c0792e276e3e7cd4f
2023-11-02T15:57:49.925550138Z     Parent ID      : 5f7bb59693824b8e
2023-11-02T15:57:49.925558544Z     ID             : 6fe8f6d99948db15
2023-11-02T15:57:49.925566228Z     Name           : api/WeatherForecast/list
2023-11-02T15:57:49.925574724Z     Kind           : Server
2023-11-02T15:57:49.925583160Z     Start time     : 2023-11-02 15:57:38.985265 +0000 UTC
2023-11-02T15:57:49.925590593Z     End time       : 2023-11-02 15:57:41.0932854 +0000 UTC
2023-11-02T15:57:49.925598869Z     Status code    : Unset
2023-11-02T15:57:49.925606684Z     Status message : 
2023-11-02T15:57:49.925614398Z Attributes:
2023-11-02T15:57:49.925622273Z      -> net.host.name: Str(otelapiexample.azurecontainerapps.io)
2023-11-02T15:57:49.925630308Z      -> http.method: Str(GET)
2023-11-02T15:57:49.925637792Z      -> http.scheme: Str(http)
2023-11-02T15:57:49.925656327Z      -> http.target: Str(/api/WeatherForecast/list)
2023-11-02T15:57:49.925664993Z      -> http.url: Str(http://otelapiexample.azurecontainerapps.io/api/WeatherForecast/list)
2023-11-02T15:57:49.925672417Z      -> http.flavor: Str(1.1)
2023-11-02T15:57:49.925680452Z      -> requestProtocol: Str(HTTP/1.1)
2023-11-02T15:57:49.925688497Z      -> http.route: Str(api/WeatherForecast/list)
2023-11-02T15:57:49.925696131Z      -> http.status_code: Int(200)
2023-11-02T15:57:49.925704116Z      -> collector.hostname: Str(collector--j38bkcz-6f84c5bb7d-sdkhp)
2023-11-02T15:57:49.925711771Z      -> azure.container_app.revision: Str(collector--j38bkcz)
2023-11-02T15:57:49.925719856Z      -> azure.container_app.name: Str(collector)
2023-11-02T15:57:49.925728021Z      -> source.blog: Str(true)
2023-11-02T15:57:49.925735806Z 	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2023-11-02T15:57:49.925756164Z 2023-11-02T15:57:49.923Z	debug	otlphttpexporter@v0.67.0/otlp.go:127	Preparing to make HTTP request	{"kind": "exporter", "data_type": "traces", "name": "otlphttp/withauth", "url": "https://url/data/v1/trace"}

[crobert-1]

Hello @dickiebowuk, can you share the configuration of the collector that you're running?

[dickiebowuk]

Here is the config. It is set to send data to both New Relic and AppDynamics CNAO platform.

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlphttp/withauth:
    auth:
      authenticator: oauth2client
    logs_endpoint: $CNAO_LOGS_ENDPOINT
    metrics_endpoint: $CNAO_METRICS_ENDPOINT
    traces_endpoint: $CNAO_TRACES_ENDPOINT
  otlp:
    endpoint: $NEW_RELIC_END_POINT
    headers:
      api-key: $NEW_RELIC_API_KEY

  logging:
    loglevel: debug

processors:
  batch:
    send_batch_size: 8192
    timeout: 10s
  attributes/collector_info:
    actions:
      - key: collector.hostname
        value: $HOSTNAME
        action: insert
      - key: azure.container_app.revision
        value: $CONTAINER_APP_REVISION
        action: insert
      - key: azure.container_app.name
        value: $CONTAINER_APP_NAME
        action: insert
      - key: source.blog
        value: "true"
        action: insert
  filter/healthcheck:
    spans:
      exclude:
        match_type: strict
        attributes:
          - Key: http.target
            Value: /health

extensions:
  oauth2client:
    client_id: $CNAO_CLIENT_ID
    client_secret: $CNAO_CLIENT_SECRET
    token_url: $CNAO_TOKEN_URL

service:
  extensions: [oauth2client]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch,filter/healthcheck,attributes/collector_info]
      exporters: [logging, otlphttp/withauth, otlp]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/withauth, otlp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/withauth, otlp]
  telemetry:
    logs:
      level: "debug"

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[atoulme]

I don't understand how to help you. I am not sure where the collector plays a role here. I suggest you provide a bug report with reproducible steps we can follow to help and assign the issue. I am closing the issue, please reopen if more work is needed.