Not sure how to get tracing ingesting to work

[uhfath]

Here is a sample of a log sent through Vector to Seq using HTTP ingesting:

{
  "@t": "2024-05-14T08:09:37.5882985Z",
  "@mt": "Sending file. Request path: '{VirtualPath}'. Physical path: '{PhysicalPath}'",
  "@tr": "c249130babf90db839df1265014b1ce2",
  "@sp": "91c53fab6829c37e",
  "VirtualPath": "/_content/Radzen.Blazor/fonts/SourceSansPro-Semibold.woff",
  "PhysicalPath": "D:\\Projects\\MED.Framework\\ThirdParty\\radzen-blazor\\Radzen.Blazor\\wwwroot\\fonts\\SourceSansPro-Semibold.woff",
  "EventId": {
    "Id": 2,
    "Name": "FileServed"
  },
  "SourceContext": "Microsoft.AspNetCore.StaticFiles.StaticFileMiddleware",
  "RequestId": "0HN3JTKN4UK22:00000007",
  "RequestPath": "/_content/Radzen.Blazor/fonts/SourceSansPro-Semibold.woff",
  "ConnectionId": "0HN3JTKN4UK22",
  "@sc": "Microsoft.AspNetCore.Hosting.HttpRequestIn",
  "@Elapsed": "00:00:00",
  "@st": "2024-05-14T08:09:37.5858124Z",
  "@ps": "0000000000000000",
  "Environment": "Development",
  "Application": "MED.Incident.UI.Host",
  "Assembly": "MED.Incident.UI.Host",
  "AssemblyVersion": "1.0.0.0"
}

And here is how it looks like in Seq:

There are a few things that I don't understand:

1. If I click Trace -> Show then only one single trace is shown in the window above. I thought there will be a series of traces. Also no matter how many times I click Trace -> Show on other entries, only the first one is shown - the window is not updated. Only disabling/enabling tracing with a button on the top helps to reset it.
2. What is the proper CLEF format to ingest @Elapsed as described here. Should I just send it as @Elapsed or there is a special name for this key?
3. How to show the traces in a hierarchical manner like it's shown in Seq docs?
4. Somehow @sc is shown as part of Properties, but was sent at the root level. Is something wrong with my log format? Same with @Elapsed.

If it helps, I've attached the log file (with just these entries):
logs.log

Can someone please explain to me how to use tracing properly? I'm sure there's something obvious I'm not getting, but I just can't figure it out.

[nblumhardt]

Hi!

1. The window will show all spans with the same trace id (@tr) - are you expecting there to be more spans in that trace?
2. @Elapsed isn't accepted on ingest; it's calculated as @t - @st
3. This falls out of (1), spans in the same trace need to have the same trace id, and the hierarchy is defined by their parent span (@ps) properties
4. @sc should be an object, like "@sc": {"name": "..."} - though it'll still appear in the properties list, just slightly differently :-)

If you're not running 2024.3, you'll need that in order for @sc to be recognized IIRC.

Hope this helps!

Nick

[uhfath]

Thank you for detailed information.

The window will show all spans with the same trace id (@tr) - are you expecting there to be more spans in that trace?

Yes. All these entries use the same @tr value. The log file attached is "filtered" to these. It's also visible in a screenshot (the search field part).

@Elapsed isn't accepted on ingest; it's calculated as @t - @st

Got it. Thanks! I've mistakenly thought that it should be also transmitted.

This falls out of (1), spans in the same trace need to have the same trace id, and the hierarchy is defined by their parent span (@ps) properties

As to @ps it's also same for these entries - it's zero actually.

@sc should be an object, like "@sc": {"name": "..."} - though it'll still appear in the properties list, just slightly differently :-)

Thanks. Will convert and take a look.

And yes, we are using 2024.3.11510 version right now.

[uhfath]

I've just recorded a small video clip of how it looks like on my machine.
Notice that trace window shows one specific entry and not all of them which are currently filtered under same @tr.

2024-05-14.14-27-44.mp4

Could this be because all entries have the same @sp ?

[liammclennan]

@uhfath @sp is the span id, so yes the behavior is undefined if many spans have the same span id.

[nblumhardt]

Just to add one more note, leaning on https://github.com/serilog-tracing/serilog-tracing for this should make everything much smoother. HTH!

[uhfath]

@liammclennan makes sense. Thanks!

@nblumhardt thanks for the link, but unfortunately if I understand correctly it works solely with Serilog's ILogger and not MEL which is required in our case. That is why we made our own enricher and some custom utility classes for activities.

[nblumhardt]

@uhfath no, SerilogTracing hooks into System.Diagnostics.ActivitySource, so if you create Activity instances in your code, you can collect them with SerilogTracing whether you are using its Serilog extensions or not. Your use case is one of the core scenarios it supports 👍 HTH!

[uhfath]

@nblumhardt interesting enough, I couldn't make it to work the first time I tried it but gave it another go after your comment and it seems to work now. Thanks!
But there is another issue now:

Notice that "Missing" part. What could it be?
The setup is quite simple in my case. The only change is this:

using var _ = new ActivityListenerConfiguration()
    .Instrument.AspNetCoreRequests()
    .Instrument.HttpClientRequests()
    .TraceToSharedLogger();

Currently we are not using Serilog's activity extensions. Just this setup and standard MEL stuff.