Setting breakpoint in managed function using lldb

[ashishnegi]

I want to set breakpoint on my function Foo in below program.

// Program.cs

using System;
using System.Threading;

namespace lldblivedebug
{
    class Program
    {
        static void Main(string[] args)
        {
            int count = 1;
            while (count > 0)
            {
                 Thread.Sleep(2000);
                 count = count + 1;
                 Foo();
            }
        }

        static void Foo()
        {
             Console.WriteLine("From foo");
        }
    }
}

I can see managed stack trace. But I am not able to set the breakpoints on function.
I tried command bpmd. I don't totally understand what should be next argument to bpmd on Linux.
bpmd <module_name> <Fully_qualified_funciton>.

If my executable name is X and code is in X.dll on linux, Is module_name in above command X.dll or X ?

Here is a sample lldb session of what I tried.

dotnet build -r ubuntu.16.04-x64
cd ~/dotnet_test/lldblivedebug/bin/Debug/netcoreapp2.0/ubuntu.16.04-x64

asnegi@asnegi-vm ~/dotnet_test/lldblivedebug/bin/Debug/netcoreapp2.0/ubuntu.16.04-x64 $ lldb-3.9 -O "plugin load ~/ryu/libsosplugin/libsosplugin.so" -O "target create lldblivedebug" -o "breakpoint set -n lldblivedebug.Program.Main"  lldblivedebug linux
(lldb) plugin load ~/ryu/libsosplugin/libsosplugin.so
(lldb) target create lldblivedebug
Current executable set to 'lldblivedebug' (x86_64).
(lldb) target create "lldblivedebug"
Current executable set to 'lldblivedebug' (x86_64).
(lldb) settings set -- target.run-args  "linux"
(lldb) breakpoint set -n lldblivedebug.Program.Main
Breakpoint 1: no locations (pending).
WARNING:  Unable to resolve breakpoint to any actual locations.
(lldb) r
Process 28055 launched: '/home/asnegi/dotnet_test/lldblivedebug/bin/Debug/netcoreapp2.0/ubuntu.16.04-x64/lldblivedebug' (x86_64)
From foo
Process 28055 stopped
* thread dotnet/coreclr#1: tid = 28055, 0x00007ffff79c3709 libpthread.so.0`__pthread_cond_timedwait + 297, name = 'lldblivedebug', stop reason = signal SIGSTOP
    frame #0: 0x00007ffff79c3709 libpthread.so.0`__pthread_cond_timedwait + 297
libpthread.so.0`__pthread_cond_timedwait:
->  0x7ffff79c3709 <+297>: movq   %rax, %r14
    0x7ffff79c370c <+300>: movl   (%rsp), %edi
    0x7ffff79c370f <+303>: callq  0x7ffff79c61c0            ; __pthread_disable_asynccancel
    0x7ffff79c3714 <+308>: movq   0x8(%rsp), %rdi

Here From foo comes from Foo function.

(lldb) sos ClrStack
OS Thread Id: 0x6d97 (1)
        Child SP               IP Call Site
00007FFFFFFFCE18 00007ffff79c3709 [HelperMethodFrame: 00007fffffffce18] System.Threading.Thread.SleepInternal(Int32)
00007FFFFFFFCF40 00007FFF7CB3661D System.Threading.Thread.Sleep(Int32)
00007FFFFFFFCF60 00007FFF7D1B592B System.Threading.Thread.Sleep(Int32)
00007FFFFFFFCF70 00007FFF7D1504C0 lldblivedebug.Program.Main(System.String[])
00007FFFFFFFD260 00007ffff6313f07 [GCFrame: 00007fffffffd260]
00007FFFFFFFD670 00007ffff6313f07 [GCFrame: 00007fffffffd670]
(lldb) target modules list
[  0] B1E8F043-DAAA-7034-9629-D6129728A989-1A19C89E                    /home/asnegi/dotnet_test/lldblivedebug/bin/Debug/netcoreapp2.0/ubuntu.16.04-x64/lldblivedebug
[  1] BD45E79E-8A08-DCE5-67A1-E9EFB2F970A9-B2DC77D8 0x00007ffff7ffa000 [vdso] (0x00007ffff7ffa000)
[  2] 8CC8D0D1-19B1-42D8-3980-0BFF71FB71E7-3AEA7BD4                    /lib/x86_64-linux-gnu/libdl.so.2
      /usr/lib/debug/lib/x86_64-linux-gnu/libdl-2.23.so
[  3] CE17E023-5422-65FC-11D9-BC8F534BB4F0-70493D30                    /lib/x86_64-linux-gnu/libpthread.so.0
[  4] CB93C881-929B-523C-01AC-EF171B52D526-1F026029                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6
[  5] DFB85DE4-2DAF-FD09-640C-8FE377D572DE-3E168920                    /lib/x86_64-linux-gnu/libm.so.6
      /usr/lib/debug/lib/x86_64-linux-gnu/libm-2.23.so
[  6] 68220AE2-C65D-65C1-B6AA-A12FA6765A6E-C2F5F434                    /lib/x86_64-linux-gnu/libgcc_s.so.1
[  7] B5381A45-7906-D279-0738-22A5CEB24C4B-FEF94DDB                    /lib/x86_64-linux-gnu/libc.so.6
      /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.23.so
[  8] 5D7B6259-5522-75A3-C17B-D4C3FD05F5A6-BF40CAA5                    /lib64/ld-linux-x86-64.so.2
[  9] FA3EC34F-FF22-C058-FE6C-120AD8466ACE-A05DCEBB                    /home/asnegi/dotnet_test/lldblivedebug/bin/Debug/netcoreapp2.0/ubuntu.16.04-x64/libhostfxr.so
[ 10] 82FA3DDE-91A5-5710-F6CD-42274FF2701E-5A2E9BE3                    /home/asnegi/dotnet_test/lldblivedebug/bin/Debug/netcoreapp2.0/ubuntu.16.04-x64/libhostpolicy.so
[ 11] FC9AA4F0-E049-E703-DA1B-0108018007F6-17295119                    /home/asnegi/.nuget/packages/runtime.linux-x64.microsoft.netcore.app/2.0.0/runtimes/linux-x64/native/libcoreclr.so
[ 12] 89C34D7A-1823-87D7-6D5C-DA1F7718F5D5-8824DFB3                    /lib/x86_64-linux-gnu/librt.so.1
      /usr/lib/debug/lib/x86_64-linux-gnu/librt-2.23.so
[ 13] 29F63E3B-24F9-5E95-5C76-E58354DB3294-944105A1                    /usr/lib/x86_64-linux-gnu/libunwind.so.8
[ 14] 13BF33CF-CE9A-8E60-360A-390912876AC2-75C8AD93                    /lib/x86_64-linux-gnu/libuuid.so.1
[ 15] 1E80512E-50EA-07A3-BEB3-6D91C30B3B33-283967AB                    /usr/lib/x86_64-linux-gnu/libunwind-x86_64.so.8
[ 16] 15AED485-5920-E5A0-FB87-91B683EB88C7-E1199260                    /lib/x86_64-linux-gnu/liblzma.so.5
[ 17] 6EF619CF-DB93-245E-64CF-A85A216BC193-BBDC2C41                    /home/asnegi/.nuget/packages/runtime.linux-x64.microsoft.netcore.app/2.0.0/runtimes/linux-x64/native/libcoreclrtraceptprovider.so
[ 18] 269756FB-4AD2-E616-3CA4-FD34A3237739-A3AA962C                    /usr/lib/x86_64-linux-gnu/liblttng-ust.so.0
[ 19] 499250C2-6B5E-9CDE-B4A1-BBD03E15A2F5-E0B9CE55                    /usr/lib/x86_64-linux-gnu/liblttng-ust-tracepoint.so.0
[ 20] C090387F-8C58-1779-4A75-0B3439405DAA-C1FA75EE                    /usr/lib/x86_64-linux-gnu/liburcu-bp.so.4
[ 21] 9A5A4438-D46B-C130-7719-71A903B29757-00C8BC25                    /usr/lib/x86_64-linux-gnu/liburcu-cds.so.4
[ 22] 92F4379E-B144-FCB9-47CD-A120F46B67A7-A38D39F9                    /home/asnegi/.nuget/packages/runtime.linux-x64.microsoft.netcore.app/2.0.0/runtimes/linux-x64/native/libclrjit.so
[ 23] F3FA2CBA-8B15-ABC5-E785-BE2B38B6F620-E4E617A4                    /home/asnegi/.nuget/packages/runtime.linux-x64.microsoft.netcore.app/2.0.0/runtimes/linux-x64/native/System.Globalization.Native.so
[ 24] 463D8B61-0702-D64A-E080-3C7DFCAA02CF-B4C6477B                    /usr/lib/x86_64-linux-gnu/libicuuc.so.55.1
[ 25] 2CC92B3E-C411-16DD-818F-3DAF7DA7CBB4-10183DD6                    /usr/lib/x86_64-linux-gnu/libicudata.so.55
[ 26] F5BE69B9-2B74-26F1-7A61-E1A1308115A9-FE73C51D                    /usr/lib/x86_64-linux-gnu/libicui18n.so.55.1
[ 27] 7B4BE8CE-1CC5-DCF8-629A-A008500A93F0-E77577A0                    /home/asnegi/.nuget/packages/runtime.linux-x64.microsoft.netcore.app/2.0.0/runtimes/linux-x64/native/System.Native.so
(lldb) bpmd lldblivedebug lldblivedebug.Program.Foo
Adding pending breakpoints...
(lldb) bpmd lldblivedebug Program.Foo
Adding pending breakpoints...
(lldb) bpmd lldblivedebug.Program.Foo
Adding pending breakpoints...
(lldb) bpmd -list
bpmd pending breakpoint list
 Breakpoint index - Location, ModuleID, Method Token
1 - lldblivedebug.Program.Foo!Program.Foo+0, 0x0000000000000000, 0x00000000
2 - lldblivedebug!Program.Foo+0, 0x0000000000000000, 0x00000000
3 - lldblivedebug!lldblivedebug.Program.Foo+0, 0x0000000000000000, 0x00000000
(lldb) c
Process 28055 resuming
From foo
From foo
Process 28055 stopped
* thread dotnet/coreclr#1: tid = 28055, 0x00007ffff79c3709 libpthread.so.0`__pthread_cond_timedwait + 297, name = 'lldblivedebug', stop reason = signal SIGSTOP
    frame #0: 0x00007ffff79c3709 libpthread.so.0`__pthread_cond_timedwait + 297
libpthread.so.0`__pthread_cond_timedwait:
->  0x7ffff79c3709 <+297>: movq   %rax, %r14
    0x7ffff79c370c <+300>: movl   (%rsp), %edi
    0x7ffff79c370f <+303>: callq  0x7ffff79c61c0            ; __pthread_disable_asynccancel
    0x7ffff79c3714 <+308>: movq   0x8(%rsp), %rdi

Q1. Why does not target modules list does not have lldblivedebug.dll in its list ?
Is [vdso] (0x00007ffff7ffa000) in above target modules list actually lldblivedebug.dll ?

Q2. I think I don't understand how to use bpmd command in lldb on Linux. Can you please explain how to pass right module name to bpmd ?

Other info :

asnegi@asnegi-vm ~/dotnet_test/lldblivedebug/bin/Debug/netcoreapp2.0/ubuntu.16.04-x64 $ dotnet --info
.NET Command Line Tools (2.0.3)

Product Information:
 Version:            2.0.3
 Commit SHA-1 hash:  eb1d5ee318

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.0.3/

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.7
  Build    : 2d61d0b043915bc948ebf98836fefe9ba942be11

[jkotas]

cc @mikem8361

[janvorli]

Q1. Why does not target modules list does not have lldblivedebug.dll in its list ?

The managed assemblies are not modules recognized by Linux. They are not loaded using the platform loader. They are not ELF files, but rather Windows style PE files.

Q2. I think I don't understand how to use bpmd command in lldb on Linux. Can you please explain how to pass right module name to bpmd ?

The right syntax is bpmd lldblivedebug.dll lldblivedebug.Program.Foo
@mikem8361 I've verified that using just the module name (lldblivedebug) doesn't work, the extension needs to be specified too. Is that intentional?

[mikem8361]

Yes, the managed assemblies don’t show in the lldb modules command unlike on Windows. And yes you need usually need to add the .dll extension depending on how the assembly was loaded. The “sos DumpDomain” command displays all the loaded assemblies. The “soshelp bpmd” has some more details on the command. (lldb) sos DumpDomain -------------------------------------- System Domain: 00007ffff6d965f0 LowFrequencyHeap: 00007FFFF6D974C8 HighFrequencyHeap: 00007FFFF6D97578 StubHeap: 00007FFFF6D97628 Stage: OPEN Name: None -------------------------------------- Shared Domain: 00007ffff6d95600 LowFrequencyHeap: 00007FFFF6D974C8 HighFrequencyHeap: 00007FFFF6D97578 StubHeap: 00007FFFF6D97628 Stage: OPEN Name: None Assembly: 000000000069eed0 [/home/mikem/coreclr/bin/Product/Linux.x64.Debug/System.Private.CoreLib.dll] ClassLoader: 000000000069EF90 Module Name 00007fff7c190400 /home/mikem/coreclr/bin/Product/Linux.x64.Debug/System.Private.CoreLib.dll

…

-------------------------------------- Domain 1: 0000000000657a60 LowFrequencyHeap: 0000000000658B50 HighFrequencyHeap: 0000000000658C00 StubHeap: 0000000000658CB0 Stage: OPEN Name: unixcorerun … Assembly: 00000000006b86a0 [/home/mikem/coreclr/bin/Product/Linux.x64.Debug/SymbolTestApp2.dll] ClassLoader: 00000000006B8760 Module Name 00007fff7c073d20 /home/mikem/coreclr/bin/Product/Linux.x64.Debug/SymbolTestApp2.dll Assembly: 00000000006c8140 [/home/mikem/coreclr/bin/Product/Linux.x64.Debug/System.Runtime.dll] ClassLoader: 00000000006BA3F0 Module Name 00007fff7c074978 /home/mikem/coreclr/bin/Product/Linux.x64.Debug/System.Runtime.dll … Assembly: 0000000000782920 [SymbolTestDll, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null] ClassLoader: 00000000007829E0 Module Name 00007fff7d9aab58 SymbolTestDll, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null For the first module, the command would be: bpmd SymbolTestApp2.dll SymbolTestApp.Program.Foo4 The second module was loaded with LoadAssembly and DumpDomain displays no .dll (just SymbolTestDll): bpmd SymbolTestDll SymbolTestDll.TestClass.Foo7

[ashishnegi]

Thanks @janvorli @mikem8361 for useful comments. I can attach to the function now.
Is there any more debugging tips there ?
I found about bpmd command from another debugging issue. Where can I see a comprehensive list of all these commands ?

[janvorli]

@ashishnegi sos help command should show all the supported ones. There is also a MSDN page on sos.dll with detailed explanation of the sos commands (https://docs.microsoft.com/en-us/dotnet/framework/tools/sos-dll-sos-debugging-extension), but some of them don't work on Unix and some of them have slightly different parameters on Unix. However, it is good enough as a guide overall.