Missing symbols for only dotnet code

[careless6666]

I write minimal dotnet app and try get trace for it

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI();
app.MapControllers();
app.MapGet("test_slowpoke", () =>
{
	long counter = 0;
	for (int i = 0; i < 1_000_000_000; i++)
	{
		counter++;
	}
	
	return "Hello world";
});

app.Run();

But in result trace you can wee missing symbols

record trace by command:

dotnet-trace collect --duration 00:00:00:20 -- WebApplication1

statistic:

dotnet-trace report WebApplication1_20230217_193155.nettrace topN -n 10


Top 10 Functions (Exclusive)                                                  Inclusive           Exclusive           
1.  Missing Symbol                                                            58.22%              58.19%              
2.  WaitHandle.WaitOneNoCheck(int32)                                          10.13%              10.13%              
3.  Thread.Sleep(int32)                                                       10.09%              10.09%              
4.  ManualResetEventSlim.Wait(int32,value class System.Threading.Cancellat    10.08%              10.08%              
5.  webapplication1!Program+<>c.<<Main>$>b__0_0()                             1.13%               1.13%               
6.  WebHost.ConfigureWebDefaults(class Microsoft.AspNetCore.Hosting.IWebHo    0.02%               0.01%               
7.  .cctor()                                                                  0.01%               0.01%               
8.  corelib!Interop+Sys.LStat(unsigned int8&,value class FileStatus&)         0%                  0%                  
9.  Http1ParsingHandler].TryTakeSingleHeader(!0,value class System.ReadOnl    0.01%               0%                  
10. RuntimeType.CreateInstanceOfT()                                           0.01%               0%        

Tell me what symbols you need to load to see the statistics

my machine
PRETTY_NAME="Ubuntu 22.04.1 LTS"
architecture x86
dotnet 7

Tagging subscribers to this area: @tommcdon
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I write minimal dotnet app and try get trace for it

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddControllers();
builder.Services.AddEndpointsApiExplorer();
builder.Services.AddSwaggerGen();

var app = builder.Build();

app.UseSwagger();
app.UseSwaggerUI();
app.MapControllers();
app.MapGet("test_slowpoke", () =>
{
	long counter = 0;
	for (int i = 0; i < 1_000_000_000; i++)
	{
		counter++;
	}
	
	return "Hello world";
});

app.Run();

But in result trace you can wee missing symbols

record trace by command:

dotnet-trace collect --duration 00:00:00:20 -- WebApplication1

statistic:

dotnet-trace report WebApplication1_20230217_193155.nettrace topN -n 10


Top 10 Functions (Exclusive)                                                  Inclusive           Exclusive           
1.  Missing Symbol                                                            58.22%              58.19%              
2.  WaitHandle.WaitOneNoCheck(int32)                                          10.13%              10.13%              
3.  Thread.Sleep(int32)                                                       10.09%              10.09%              
4.  ManualResetEventSlim.Wait(int32,value class System.Threading.Cancellat    10.08%              10.08%              
5.  webapplication1!Program+<>c.<<Main>$>b__0_0()                             1.13%               1.13%               
6.  WebHost.ConfigureWebDefaults(class Microsoft.AspNetCore.Hosting.IWebHo    0.02%               0.01%               
7.  .cctor()                                                                  0.01%               0.01%               
8.  corelib!Interop+Sys.LStat(unsigned int8&,value class FileStatus&)         0%                  0%                  
9.  Http1ParsingHandler].TryTakeSingleHeader(!0,value class System.ReadOnl    0.01%               0%                  
10. RuntimeType.CreateInstanceOfT()                                           0.01%               0%        

Tell me what symbols you need to load to see the statistics

my machine
PRETTY_NAME="Ubuntu 22.04.1 LTS"
architecture x86
dotnet 7

Author:	careless6666
Assignees:	-
Labels:	area-System.Diagnostics, untriaged
Milestone:	-

[hoyosjs]

cc: @davmason @dramos020

[davmason]

Hi @careless6666,

I just tried to run your program and could not reproduce the missing symbol issue, although that is probably because I have symbols for dotnet set up.

I get this output:

Top 10 Functions (Exclusive)                                                  Inclusive           Exclusive

1.  il!Interop+Kernel32.GetQueuedCompletionStatus(int,unsigned int32&,unsi    35.75%              35.75%

2.  PortableThreadPool+IOCompletionPoller.Poll()                              21.05%              21.05%

3.  WaitHandle.WaitOneNoCheck(int32)                                          10.46%              10.46%

4.  Thread.Sleep(int32)                                                       10.42%              10.42%

5.  ManualResetEventSlim.Wait(int32,value class System.Threading.Cancellat    10.41%              10.41%

6.  test_webapp!Program+<>c.<<Main>$>b__0_0()                                 1.15%               1.15%

7.  __Canon].get_Values()                                                     0.06%               0.06%

8.  test_webapp!Program.<Main>$(class System.String[])                        10.55%              0.03%

9.  String.Ctor(wchar*)                                                       0.02%               0.02%

10. RuntimeTypeHandle.GetTypeByName(class System.String,bool,bool,value cl    0.01%               0.01%

So it seems like you are missing symbols for il!Interop and PortableThreadPool, which are both in the dotnet runtime.

The topN command current does not allow you to set the symbol path directly, can you try setting _NT_SYMBOL_PATH=srv*c:\DownstreamStore*https://msdl.microsoft.com/download/symbols as described here?

[careless6666]

Sorry, but you give me a link to the documentation for Windows. I have dotnet-symbol installed on my linux, I try to download all symbols for my trace by

dotnet-symbol WebApplication1_20230217_192413.nettrace --microsoft-symbol-server

When i check my cache symbol cache folder and nothing changed

~/.dotnet/symbolcache$ ls -la
total 32
drwxrwxr-x  8 alex alex 4096 jan 18 15:39 .
drwxrwxr-x 10 alex alex 4096 feb 27 09:37 ..
drwxrwxr-x  3 alex alex 4096 jan 18 15:38 libclrjit.so
drwxrwxr-x  4 alex alex 4096 jan 18 15:39 libmscordaccore.so
drwxrwxr-x  3 alex alex 4096 jan 18 15:39 microsoft.extensions.hosting.abstractions.dll
drwxrwxr-x  3 alex alex 4096 jan 18 15:39 microsoft.extensions.hosting.abstractions.pdb
drwxrwxr-x  3 alex alex 4096 jan 18 15:39 system.private.corelib.dll
drwxrwxr-x  4 alex alex 4096 jan 18 15:39 system.private.corelib.pdb

I setup _NT_SYMBOL_PATH

export _NT_SYMBOL_PATH=/home/alex/.dotnet/symbolcache
echo $_NT_SYMBOL_PATH
/home/alex/.dotnet/symbolcache

But it did not help

I tried downloading symbols manually from the site(https://msdl.microsoft.com/download/symbols), but it seems that this method is currently not available from the browser

[davmason]

I don't believe dotnet-symbol supports trace files as an argument, you would need to either pass in a dump file of your app or the dotnet binaries (libcoreclr.so, System.Private.Corelib.dll, etc) that would be at $DOTNET_ROOT/shared/Microsoft.NETCore.App/<version>/

[careless6666]

I guess the problem is still in me, although I don't know what I'm doing wrong

I download all symbols

for i in `ls` ;     do   echo     `dotnet-symbol $i -o /home/alex/.dotnet/symbolcache`  ;     done

ls /home/alex/.dotnet/symbolcache

createdump.dbg                                         
System.IO.FileSystem.Primitives.pdb                 
System.Resources.ResourceManager.pdb                                     
system.io.filesystem.watcher.ni.r2rmap              
system.resources.writer.ni.r2rmap
libclrgc.so.dbg                                        
System.IO.FileSystem.Watcher.ni.r2rmap              
System.Resources.Writer.ni.r2rmap
libclrjit.so                                           
system.io.filesystem.watcher.pdb                    
system.resources.writer.pdb
libclrjit.so.dbg                                       
System.IO.FileSystem.Watcher.pdb                    
System.Resources.Writer.pdb
libcoreclr.so.dbg                                      
system.io.isolatedstorage.ni.r2rmap                 
system.runtime.compilerservices.unsafe.pdb
libcoreclrtraceptprovider.so.dbg                       
System.IO.IsolatedStorage.ni.r2rmap                 
System.Runtime.CompilerServices.Unsafe.pdb
libhostpolicy.so.dbg                                   
system.io.isolatedstorage.pdb                       
system.runtime.compilerservices.visualc.ni.r2rmap
libmscordaccore.so                                     
System.IO.IsolatedStorage.pdb                       
System.Runtime.CompilerServices.VisualC.ni.r2rmap
libmscordaccore.so.dbg                                 
system.io.memorymappedfiles.ni.r2rmap               
system.runtime.compilerservices.visualc.pdb
libmscordbi.so.dbg                                     
System.IO.MemoryMappedFiles.ni.r2rmap               
System.Runtime.CompilerServices.VisualC.pdb
libSystem.Globalization.Native.so.dbg                  
system.io.memorymappedfiles.pdb                     
system.runtime.extensions.pdb
libSystem.IO.Compression.Native.so.dbg                 
System.IO.MemoryMappedFiles.pdb                     
System.Runtime.Extensions.pdb
libSystem.Native.so.dbg                                
system.io.pdb                                       
system.runtime.handles.pdb
libSystem.Net.Security.Native.so.dbg                   
System.IO.pdb                                       
System.Runtime.Handles.pdb
libSystem.Security.Cryptography.Native.OpenSsl.so.dbg  
system.io.pipes.accesscontrol.ni.r2rmap             
system.runtime.interopservices.javascript.ni.r2rmap
microsoft.csharp.ni.r2rmap                             
System.IO.Pipes.AccessControl.ni.r2rmap             
System.Runtime.InteropServices.JavaScript.ni.r2rmap
Microsoft.CSharp.ni.r2rmap                             
system.io.pipes.accesscontrol.pdb    

... long list        

It seems I downloaded all possible pdb but the result is the same, missing symbols

[davmason]

It may not be your issue, we didn't provide an option to specify the symbol path as part of dotnet-trace report topN, which was an oversight on our part. I was trying to see if there was a way we could work around it with _NT_SYMBOL_PATH but it looks like we can't.

Would you be willing to try a private build of dotnet-trace with changes? This version of dotnet-trace has the additional argument --additional-symbol-paths that you can provide a comma separated list of paths to look for symbols in. https://github.com/davmason/diagnostics/releases/tag/v1.0.0-private-dotnet-trace

You should be able to download, unzip and run it on an existing trace

[careless6666]

I tried everything as in that branch, checking that the path is the one that should be, but the result is the same, can you attach your symbols for comparison, what is missing?
My symbols
https://drive.google.com/file/d/1bOHE342bHJCYLV53FadT1BphPKInswhF/view?usp=sharing
value from symbol paths variable:

;SRV*https://msdl.microsoft.com/download/symbols;SRV*https://nuget.smbsrc.net;SRV*https://referencesource.microsoft.com/symbols;/home/alex/.dotnet/symbolcache

and receive same result

[davmason]

Can you try with the private dotnet-trace and without the srv*? Like dotnet-trace report <trace> topN --additionalsymbolpaths https://nuget.smbsrc.net;https://referencesource.microsoft.com/symbols;/home/alex/.dotnet/symbolcache

[careless6666]

I took the code from your branch and replaced the default source with https://nuget.smbsrc.net;https://referencesource.microsoft.com/symbols
passed his characters through the parameter

Maybe it would be better if I wrap my example in docker and write here code of docker file?

[davmason]

@careless6666 did you try using the --additionalsymbolpaths argument and passing in your /home/alex/.dotnet/symbolcache path?

[careless6666]

Yes of course