Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot debug go plugins on mac #1628

Open
nd opened this issue Jul 22, 2019 · 28 comments
Open

Cannot debug go plugins on mac #1628

nd opened this issue Jul 22, 2019 · 28 comments

Comments

@nd
Copy link
Contributor

nd commented Jul 22, 2019

  1. What version of Delve are you using (dlv version)?
Delve Debugger
Version: 1.2.0
Build: cb658772972d74042ba75f006ac14023e552c7cd
  1. What version of Go are you using? (go version)?
go version go1.12.6 darwin/amd64
  1. What operating system and processor architecture are you using?
darwin/amd64
  1. What did you do?

I'm trying to debug a simple go plugin from https://github.com/vladimirvivien/go-plugin-example.

I've build plugins using the command:

go build -gcflags "all=-N -l" -x -v -buildmode=plugin -o eng/eng.so eng/greeter.go

I'm debugging the code calling the plugin (https://github.com/vladimirvivien/go-plugin-example/blob/master/greeter.go).

Once debugger reaches the breakpoint on line 58, I'm adding a breakpoint at eng/greeter.go:8 and continue or trying to step into the call.

  1. What did you expect to see?

Adding a breakpoint inside the plugin succeeds. If I continue, debugger hits the breakpoint inside the plugin. Step into the call in plugin also works.

  1. What did you see instead?

Create breakpoint call fails with the "could not find file /Users/nd/go/src/github.com/vladimirvivien/go-plugin-example/eng/greeter.go" error. Step into plugin call also fails.

Debug log: https://gist.github.com/nd/feb71212f6ef9dcfbb33c4293e18f146

Debug works fine on linux.

@juhwany
Copy link

juhwany commented Jul 22, 2019

Debugging plugin on mac is really needed for Golang developers. Hope delve supports it :)

@aarzilli
Copy link
Member

I didn't try implementing this because it would be affected by golang/go#25841 at least. I also vaguely remembering some versions of go not producing any dwarf at all for macOS. What's the output of readelf -h -S path_to_a_go_plugin.so?

@nd
Copy link
Contributor Author

nd commented Jul 23, 2019

It returns an error:

readelf: Error: Not an ELF file - it has the wrong magic bytes at the start

@aarzilli
Copy link
Member

It returns an error:

Right, of course, because they aren't elfs. My bad. I guess the right one would be otool -l or something like that. Or gobjdump -h ... if it's installed.

@nd
Copy link
Contributor Author

nd commented Jul 23, 2019

otool -l output:

eng/eng.so:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
 0xfeedfacf 16777223          3  0x00           6    13       1984 0x00100004
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 392
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x00000000000dd000
  fileoff 0
 filesize 905216
  maxprot 0x00000007
 initprot 0x00000005
   nsects 4
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000002040
      size 0x00000000000dabd0
    offset 8256
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x00000000000dcc10
      size 0x0000000000000156
    offset 904208
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 6 (size of stubs)
Section
  sectname __stub_helper
   segname __TEXT
      addr 0x00000000000dcd68
      size 0x000000000000024a
    offset 904552
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x00000000000dcfb4
      size 0x0000000000000048
    offset 905140
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 1112
  segname __DATA
   vmaddr 0x00000000000dd000
   vmsize 0x0000000000122000
  fileoff 905216
 filesize 1064960
  maxprot 0x00000007
 initprot 0x00000003
   nsects 13
    flags 0x0
Section
  sectname __nl_symbol_ptr
   segname __DATA
      addr 0x00000000000dd000
      size 0x0000000000000010
    offset 905216
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 57 (index into indirect symbol table)
 reserved2 0
Section
  sectname __got
   segname __DATA
      addr 0x00000000000dd010
      size 0x0000000000001500
    offset 905232
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 59 (index into indirect symbol table)
 reserved2 0
Section
  sectname __la_symbol_ptr
   segname __DATA
      addr 0x00000000000de510
      size 0x00000000000001c8
    offset 910608
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000007
 reserved1 731 (index into indirect symbol table)
 reserved2 0
Section
  sectname __mod_init_func
   segname __DATA
      addr 0x00000000000de6d8
      size 0x0000000000000008
    offset 911064
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000009
 reserved1 0
 reserved2 0
Section
  sectname __rodata
   segname __DATA
      addr 0x00000000000de6e0
      size 0x0000000000052d58
    offset 911072
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __typelink
   segname __DATA
      addr 0x0000000000131440
      size 0x0000000000001434
    offset 1250368
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __itablink
   segname __DATA
      addr 0x0000000000132878
      size 0x00000000000000a0
    offset 1255544
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __gosymtab
   segname __DATA
      addr 0x0000000000132918
      size 0x0000000000000000
    offset 1255704
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __gopclntab
   segname __DATA
      addr 0x0000000000132920
      size 0x000000000009a6de
    offset 1255712
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __noptrdata
   segname __DATA
      addr 0x00000000001cd000
      size 0x000000000000cbdc
    offset 1888256
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __data
   segname __DATA
      addr 0x00000000001d9be0
      size 0x0000000000006ad0
    offset 1940448
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x00000000001e06c0
      size 0x000000000001b830
    offset 0
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __noptrbss
   segname __DATA
      addr 0x00000000001fbf00
      size 0x0000000000002518
    offset 0
     align 2^5 (32)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x00000000001ff000
   vmsize 0x0000000000072000
  fileoff 1970176
 filesize 465632
  maxprot 0x00000007
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 3
          cmd LC_ID_DYLIB
      cmdsize 112
         name /var/folders/qv/lg43663s45jbmrkjvy64zh5r0000gn/T/go-build695961820/b001/exe/a.out.so (offset 24)
   time stamp 1 Thu Jan  1 01:00:01 1970
      current version 0.0.0
compatibility version 0.0.0
Load command 4
            cmd LC_DYLD_INFO_ONLY
        cmdsize 48
     rebase_off 1970176
    rebase_size 21536
       bind_off 1991712
      bind_size 65752
  weak_bind_off 0
 weak_bind_size 0
  lazy_bind_off 2057464
 lazy_bind_size 1080
     export_off 2058544
    export_size 43032
Load command 5
     cmd LC_SYMTAB
 cmdsize 24
  symoff 2105928
   nsyms 9259
  stroff 2257224
 strsize 178584
Load command 6
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 6646
     iextdefsym 6646
     nextdefsym 2555
      iundefsym 9201
      nundefsym 58
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 2254072
  nindirectsyms 788
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 7
     cmd LC_UUID
 cmdsize 24
    uuid 0CD073D4-4257-345E-8157-2ECBC871E244
Load command 8
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.12
      sdk 10.12
Load command 9
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1238.60.2
compatibility version 1.0.0
Load command 11
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 2101576
 datasize 4352
Load command 12
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 2105928
 datasize 0

@aarzilli
Copy link
Member

As you can see, no debug sections. Now that I think about it I knew this because I noticed when I changed the linker to make it able to emit debug symbols for plugins.

@juhwany
Copy link

juhwany commented Jul 31, 2019

@aarzilli

First of all, I don't know how debugger works well. But It seems that it's impossible to debug golang plugin at this moment. Because as you said, the built plugin doesn't contain debugging symbols at all.
Is this golang issue? Should I wait for new version of golang?

@akamensky
Copy link

This is a major obstacle for me as I am working on same code base that uses plugins and I often switch between Linux and Mac while working on this.

Meantime I believe lack of debugging symbols in plugins on Darwin is still an issue in Go golang/go#27502

@erikdw
Copy link

erikdw commented Dec 30, 2019

@aarzilli : Thanks for your diligence in pushing this support forward. It's nice that golang/go#27502 has now been fixed, but it seems like the ability to debug go plugins on Mac will still be blocked by golang/go#25841, according to:

Given this, perhaps you can revisit your comment in golang/go#25841, where you responded that it was not important? Seems that @heschik was willing to dig into that issue before.

@erikdw
Copy link

erikdw commented Dec 30, 2019

I discovered that the fix for golang/go#27502 is present in go1.14beta1, so I installed that version:

go get golang.org/dl/go1.14beta1
go1.14beta1 download

Then I rebuilt the binaries from the example project:

go1.14beta1 build -gcflags 'all=-N -l' -buildmode=plugin -o eng/eng.so eng/greeter.go
go1.14beta1 build -gcflags 'all=-N -l' -o greeter greeter.go

I then used otool -l to validate that the eng.so binary had the DWARF debug sections.

✗ otool -l eng/eng.so | grep debug
  sectname __zdebug_line
  sectname __zdebug_pubname
  sectname __zdebug_loc
  sectname __zdebug_ranges
  sectname __zdebug_pubtype
  sectname __zdebug_aranges
  sectname __zdebug_info
  sectname __zdebug_frame
  sectname __zdebug_abbrev
  sectname __zdebug_str

So I got my hopes up, but when I tried debugging the greeter binary I hit the same problem.

✗ dlv --check-go-version=false exec ./greeter -- english
Type 'help' for list of commands.
(dlv) b greeter.go:58
Breakpoint 1 set at 0x40f33ac for main.main() ./greeter.go:58
(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x40f33ac)
    53:			fmt.Println("unexpected type from module symbol")
    54:			os.Exit(1)
    55:		}
    56:	
    57:		// 4. use the module
=>  58:		greeter.Greet()
    59:	
    60:	}
(dlv) b eng/greeter.go:8
Command failed: Location "eng/greeter.go:8" not found
(dlv) s
Stopped at: 0x7ce92d0
=>no source available
(dlv) s
Hello Universe
Process has exited with status 0
(dlv) 

@aarzilli
Copy link
Member

I think it would behoove you to comment yourself on that thread, your first hand account of why you need it would be more persuasive than my second hand perception that some people might need it.

@erikdw
Copy link

erikdw commented Dec 30, 2019

@aarzilli : I've put in a comment on that issue. But frankly I'm too ignorant about this topic to understand what the issues are here, and how exactly golang/go#25841 is blocking this issue.

e.g., I don't understand how you obtained the output that you cite, they look like they came from running readelf, but the ticket is about Mac. So I'm guessing maybe you targeted GOOS=linux in the build. And if so then I'm not seeing how that relates to this issue.

Furthermore, I seem to be seeing at least 2 different problems in my example above, but I'm not sure if they are symptoms of one underlying cause.

cannot set the breakpoint

On Mac, where it fails in this scenario, it looks like this:

(dlv) b eng/greeter.go:8
Command failed: Location "eng/greeter.go:8" not found

On linux where it works it looks like this:

(dlv) b eng/greeter.go:8
Breakpoint 2 set at 0x7f7d8c5c838f for plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:8

cannot step into the plugin code

On Mac. where it fails in this scenario, it looks like this:

(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x40f33ac)
    53:			fmt.Println("unexpected type from module symbol")
    54:			os.Exit(1)
    55:		}
    56:	
    57:		// 4. use the module
=>  58:		greeter.Greet()
    59:	
    60:	}
(dlv) s
Stopped at: 0xcde92d0
=>no source available

On linux where it works it looks like this:

(dlv) c
> main.main() ./greeter.go:58 (hits goroutine(1):1 total:1) (PC: 0x533921)
    53:			fmt.Println("unexpected type from module symbol")
    54:			os.Exit(1)
    55:		}
    56:	
    57:		// 4. use the module
=>  58:		greeter.Greet()
    59:	
    60:	}
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.(*greeting).Greet() <autogenerated>:1 (PC: 0x7f972c3cf421)
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:7 (PC: 0x7f972c3cf381)
     2:	
     3:	import "fmt"
     4:	
     5:	type greeting string
     6:	
=>   7:	func (g greeting) Greet() {
     8:		fmt.Println("Hello Universe")
     9:	}
    10:	
    11:	// exported
    12:	var Greeter greeting
(dlv) s
> plugin/unnamed-380bc3019cae9f99f0f646d134e7c21e1405b74f.greeting.Greet() ./eng/greeter.go:8 (PC: 0x7f972c3cf38f)
     3:	import "fmt"
     4:	
     5:	type greeting string
     6:	
     7:	func (g greeting) Greet() {
=>   8:		fmt.Println("Hello Universe")
     9:	}
    10:	
    11:	// exported
    12:	var Greeter greeting

@aarzilli
Copy link
Member

aarzilli commented Jan 2, 2020

e.g., I don't understand how you obtained the output that you cite, they look like they came from running readelf

they are from objdump --dwarf, the GNU version, not the apple version.

@erikdw
Copy link

erikdw commented Jan 2, 2020

Thanks for the response. Hmm, for some reason gobjdump (GNU objdump) doesn't show any details when run against a Mac-targeted binary. Maybe I'm missing some necessary flags for the gobjdump binary?

✗ go build -gcflags='-N -l' -buildmode pie -o greeter greeter.go
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter

greeter:     file format mach-o-x86-64

Whereas when I crosscompile the binary for being run on linux (again from Mac), gobjdump --dwarf shows all kinds of fun dwarf details.

✗ GOOS=linux GOARCH=amd64 go build -gcflags '-N -l' -o greeter greeter.go
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter | head     

greeter:     file format elf64-x86-64

Contents of the .zdebug_abbrev section:

  Number TAG (0x0)
   1      DW_TAG_compile_unit    [has children]
    DW_AT_name         DW_FORM_string
    DW_AT_language     DW_FORM_data1
    DW_AT_stmt_list    DW_FORM_sec_offset
...

NOTE: I wasn't able to run this linux-targeted build in -buildmode pie, per the errors below:

✗ GOOS=linux GOARCH=amd64 go build -gcflags '-N -l' -buildmode pie -o greeter greeter.go
# command-line-arguments
loadinternal: cannot find runtime/cgo
/usr/local/Cellar/go/1.13.5/libexec/pkg/tool/darwin_amd64/link: running clang failed: exit status 1
ld: unknown option: -z
clang: error: linker command failed with exit code 1 (use -v to see invocation)

@aarzilli
Copy link
Member

aarzilli commented Jan 2, 2020

It's probably because of dwarf compression, you probably have to disable it with -ldflags='-compressdwarf=false'

@erikdw
Copy link

erikdw commented Jan 2, 2020

@aarzilli : woot! That was it. Thanks for teaching me something.

✗ go build -gcflags='-N -l' -ldflags='-compressdwarf=false' -buildmode pie -o greeter greeter.go 
✗ /usr/local/Cellar/binutils/2.33.1/bin/gobjdump --dwarf greeter                      

greeter:     file format mach-o-x86-64

Raw dump of debug contents of section .debug_line:

  Offset:                      0x0
  Length:                      1288
  DWARF Version:               2
...

@edwingeng
Copy link

edwingeng commented Jul 24, 2021

After a long waiting, I walked around the problem by adding a static-linking mode to my plugin builder. With https://github.com/edwingeng/hotswap, you can switch freely between the dynamic-linking mode and the static-linking mode. Debugging a plugin under Mac is no longer a trouble to me.

P.S. Don't miss the --staticLinking argument.

@FloThinksPi
Copy link

@edwingeng debugging a go plugin should still be a issue for you. When i read trough your project you generate some wrapper code to compile code that was written as go plugin into the main binary ?
Correct me if i am wrong but this is not what i would call a go plugin, for me its something compiled with buildmode=plugin that gets loaded at runtime.
So still as long as this issue in golang/go#25841 is not fixed by apple its hard to get this working. Alltough i saw commits from @aarzilli providing a workaround in delve #2374 but i am not sure if this fully gets debugging go plugins on macs working. Maybe @aarzilli can comment here on that :>

@edwingeng
Copy link

edwingeng commented Jul 24, 2021

When i read trough your project you generate some wrapper code to compile code that was written as go plugin into the main binary ?

Yes.

for me its something compiled with buildmode=plugin that gets loaded at runtime.

If --staticLinking is NOT specified when building a plugin, it is still a perfect plugin, which is not debuggable on Mac. hotswap helps me debug the code in a plugin. It is a problem solver, not a curer:)

--staticLinking is a per build choice.

@rstcruzo
Copy link

@FloThinksPi golang/go#25841 seems to be fixed, should it be working now? I still get errors trying to debug a plugin in macos with delve.

@aarzilli
Copy link
Member

Delve's side of this needs to be implemented. At the moment I don't have access to a working VM (and none of the builders have a version of macos that will receive the updated version of xcode) so I'm not going to do it any time soon, if someone else wants to do it: it should be fairly simple, just add a call to getLoadedDynamicLibraries at the end of ContinueOnce and call bi.AddImage for all the dynamic libraries.

@rstcruzo
Copy link

rstcruzo commented May 23, 2023

I guess I'll give it a try since no one more experienced with delve and go is available.

@aarzilli Please give me a hand with this. I just tried your suggestion:

  1. Added the following code right before the return statement of (*gbpProcess).ContinueOnce:

	images, err := p.conn.getLoadedDynamicLibraries()
	if err != nil {
		return nil, stopReason, fmt.Errorf("could not load dynamic libraries %s", err)
	}
	for _, image := range images {
		p.bi.AddImage(image.Pathname, image.LoadAddress)
	}

  1. go install
  2. Still getting could not find statement at /Users/rodrigo/testgo/plugin.go:8, please use a line with a statement

There must be something I'm missing.

Edit: I'm trying to use delve with dap.

Edit 2: Looks like AddImage is returning an error:

invalid magic number in record at byte 0x0

(The code I'm using to test this is just the example in the official docs https://pkg.go.dev/plugin)

@aarzilli
Copy link
Member

@rstcruzo presumably it's coming from loadBinaryInfoMacho, you should check that it can actually load the plugin binary. Could be something in go's standard library.

@aarzilli
Copy link
Member

I checked out the latest version of macOS (13.4) and it still has a bugged version of dsymutil.

@rstcruzo
Copy link

@aarzilli I updated my macbook to macOS Sonoma. Is there a command I can run to check if dsymutil was updated?

@aarzilli
Copy link
Member

No, build something and check that the debug_frame section is correct (you could probably find out from the version of llvm but I don't know which version has the fix).

@aarzilli
Copy link
Member

FYI Sonoma (or rather the version of xcode that ships with sonoma) still has an old version of dsymutil.

@shettyh
Copy link

shettyh commented Jun 8, 2024

Still not able to debug plugins, is there any workaround for this ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants