If users specify `OTEL_EXPORTER_OTLP_ENDPOINT`, don't overwrite it with Aspire's default OTLP endpoint

[Aaronontheweb]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe the problem.

Related: practical-otel/opentelemetry-aspire-collector#1

I just went through a lot of trouble debugging this in one of our own applications and made a YouTube video about it, but TL;DR; .NET Aspire 9.4 will front-run the OTEL_EXPORTER_OTLP_ENDPOINT and overwrite it with its own dashboard OTLP ingestion endpoint at the last minute, even though it looks like it's set correctly inside the .NET Aspire resources view in the first place.

Short video I made demo-ing this and a suggested work-around (using custom environment variables and then explicitly using them in the OTEL pipeline): https://www.youtube.com/watch?v=dCKyNCdtaVc

Describe the solution you'd like

if a user is going through the trouble of explicitly specifying OTEL_EXPORTER_OTLP_ENDPOINT inside the Aspire AppHost, Aspire shouldn't front-run that and substitute its own values. I've seen the solutions used in projects like https://github.com/dotnet/aspire-samples/tree/main/samples/Metrics and that seems... Like a tremendous amount of work on the part of the user for what amounts to setting an environment variable.

It'd be better if Aspire just didn't try to overwrite the OTEL_EXPORTER_OTLP_ENDPOINT value if it was already specified here, rather than requiring a Manhattan Project's worth of Aspire internals knowledge or a third party plugin implementing the same.

Additional context

No response

[JamesNK]

I'm not able to replicate this.

OTLP environment variables are set when a project resource is created. Because these environment variable annotations are set when the project is created, they're applied before environment variables added by the user.

For example, if someone specifies OTEL_EXPORTER_OTLP_ENDPOINT themselves then that is what is used:

builder.AddProject<Projects.BrowserTelemetry_Web>("web")
    .WithExternalHttpEndpoints()
    .WithEnvironment("OTEL_EXPORTER_OTLP_ENDPOINT", "http://localhost:8081");

Result is the user defined variable is used:

Which is confirmed to be passed to the resource by logging the OTEL_EXPORTER_OTLP_ENDPOINT env var value to the console:

I think the only way the user defined value wouldn't be used is if the env var annotation was inserted before other annotations. You can do that, but it requires some ugly code:

var p = builder.AddProject<Projects.BrowserTelemetry_Web>("web")
    .WithExternalHttpEndpoints();
p.Resource.Annotations.Insert(0, new EnvironmentCallbackAnnotation(c => c.EnvironmentVariables["OTEL_EXPORTER_OTLP_ENDPOINT"] = "http://localhost:8081"));

Result is the Aspire specified value is used over the user value:

We could update Aspire to only set these values if they're not already set, but given env vars are always applied in a order, and default behavior is user setting with WithEnvironment already take prescence over defaults, I don't think that's nessessary.

[Aaronontheweb]

var prometheus = builder.AddContainer("prometheus", "prom/prometheus", "v3.2.1")
    .WithBindMount("./prometheus", "/etc/prometheus", isReadOnly: true)
    .WithArgs("--web.enable-otlp-receiver", "--config.file=/etc/prometheus/prometheus.yaml")
    .WithHttpEndpoint(port: 9090, targetPort: 9090, name: "http");

var grafana = builder.AddContainer("grafana", "grafana/grafana")
    .WithBindMount("./grafana/config", "/etc/grafana", isReadOnly: true)
    .WithBindMount("./grafana/dashboards", "/var/lib/grafana/dashboards", isReadOnly: true)
    .WithEnvironment("PROMETHEUS_ENDPOINT", prometheus.GetEndpoint("http"))
    .WithHttpEndpoint(port: 3000, targetPort: 3000, name: "http")
    .WithLifetime(ContainerLifetime.Session); // Force fresh container each run

var otelCollector = builder.AddOpenTelemetryCollector("otelcollector", "./otel_collector/config.yaml")
    .WithEnvironment("SEQ_ENDPOINT", $"{seq.GetEndpoint("http")}/ingest/otlp")
    .WithEnvironment("PROMETHEUS_ENDPOINT", $"{prometheus.GetEndpoint("http")}/api/v1/otlp");

#endregion

#region Azure Discovery Setup

var azure = builder.AddAzureStorage("azure");

if (builder.Environment.IsDevelopment())
    azure.RunAsEmulator();

var azureTables = azure.AddTables("azure-tables");

#endregion

builder.AddProject<Projects.Petabridge_Phobos_Web>("phobos-web")
    .WithHttpsEndpoint(port: 1881)
    .WithHttpEndpoint(port: 1880)
    .WithEndpoint(name: "remoting", env: "AkkaOptions__Remote__Port")
    .WithEndpoint(name:"management", env: "AkkaOptions__Management__Port")
    .WithEndpoint(name: "pbm", env: "AkkaOptions__Pbm__Port")
    .WithEnvironment("ASPNETCORE_ENVIRONMENT", "Development")
    .WithEnvironment("DOTNET_ENVIRONMENT", "Development")
    .WithEnvironment("OTEL_EXPORTER_OTLP_ENDPOINT", otelCollector.GetEndpoint("grpc"))
    .WithReference(azureTables)
    .WithReference(seq)
    .WaitFor(prometheus)
    .WaitFor(grafana)
    .WaitFor(seq)
    .WaitFor(azureTables)
    .WithReplicas(3);

That is the code that produced the original screenshot I included.

[Aaronontheweb]

You can also try git clone && dotnet run https://github.com/practical-otel/opentelemetry-aspire-collector/ - which will also reproduce this issue.

[Aaronontheweb]

And here's my full sample, which doesn't use any of the lifecycle hooks / etc that are in the practical OTEL sample - all it does is inject the OTLP variables using the resolved endpoint of the collector. This does not work.

https://github.com/petabridge/Petabridge.Phobos.Web/tree/master/src/Petabridge.Phobos.Web.Aspire

[JamesNK]

You can also try git clone && dotnet run practical-otel/opentelemetry-aspire-collector - which will also reproduce this issue.

It appears to work for me. The only change I made was to write the env var in the apiservice start up.

Value overriden to be the OTEL collectors:

Extra startup logging in apiservice:

Confirming that OTEL collector endpoint was passed to apiservice:

Telemetry from apiservice that presumably is being sent via the collector:

[Aaronontheweb]

what change is that exactly?

[JamesNK]

Extra startup logging in apiservice:

[Aaronontheweb]

how do you explain the totally different results both Martin and I got?

[Aaronontheweb]

I didn't include this with my original screenshot, but here's what I got when running it - both of these images belong to the same run

[JamesNK]

I don't know. There isn't anything special about the OTEL_EXPORTER_OTLP_ENDPOINT environment variable other than it is set with an initial value when the resource is created.

@karolz-ms Do you have any ideas as to how one env var value could be reported in the dashboard, and another is actually set on the process?

[karolz-ms]

Works for me as well.

There is a comment on the YouTube video that says

This was to do with the eventing model changes, and one of the reasons I've moved it to the CommunityToolkit. I'll backport the code later today

So, external issue that should be fixed now?

[Aaronontheweb]

I'm having trouble recreating the problem even with my original sample (Petabridge.Phobos.Web), but running on a different machine (Ubuntu 24.04) - what I can tell you though is that the issue I had in #11298 (comment) was persistent on the Windows 11 machine where I first ran into the problem. I was also able to reproduce the same issue using https://github.com/practical-otel/opentelemetry-aspire-collector/ on that same machine, also persistently for an entire work-day.

I'll try reverting my work-around and see if I can make the issue happen again - I'm not familiar at all with Aspire's internals - is there a cache structure or something not-represented in code I should be looking for on the file system or in the environment that could explain this?

The other thing that was weird I should mention: the first time I launched my reproduction, everything worked. It was only after I started trying to film a totally different YouTube video when I re-launched the demo app again that the Aspire dashboard worked but the collector didn't - and it stayed that way.