cmd/link: bad debug_frame section on macOS with buildmode=pie [Debugging]

[aarzilli]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.10 darwin/amd64

(also reproduces with go1.11 built from tip)

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/vagrant/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/vagrant/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rm/ywx69srx1kq_dtjz2ynznn000000gp/T/go-build680864595=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Build any program with go build -gcflags='-N -l' -buildmode pie

What did you expect to see?

debug_info has an entry for main.main that looks like this

 <1><40cda>: Abbrev Number: 2 (DW_TAG_subprogram)
    <40cdb>   DW_AT_name        : (indirect string, offset: 0x29120): main.main
    <40cdf>   DW_AT_low_pc      : 0x100090530
    <40ce7>   DW_AT_high_pc     : 0x100090636
    <40cef>   DW_AT_frame_base  : 1 byte block: 9c      (DW_OP_call_frame_cfa)
    <40cf1>   DW_AT_decl_file   : 0x1
    <40cf5>   DW_AT_external    : 1

but nothing in debug_frame covers anywhere close to the range 0x100090530...0x100090636, for example the first FDE is:

00000014 000000000000002c 00000000 FDE cie=00000000 pc=0000000000002600..0000000000002737
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 23 to 0000000000002617
  DW_CFA_def_cfa_offset_sf: 48
  DW_CFA_advance_loc1: 92 to 0000000000002673
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 1 to 0000000000002674
  DW_CFA_def_cfa_offset_sf: 48
  DW_CFA_advance_loc1: 124 to 00000000000026f0
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 1 to 00000000000026f1
  DW_CFA_def_cfa_offset_sf: 48
  DW_CFA_advance_loc: 60 to 000000000000272d
  DW_CFA_def_cfa_offset_sf: 8
  DW_CFA_advance_loc: 9 to 0000000000002736
  DW_CFA_nop

It seems to me that this is wrong but I could be missing something.

[bcmills]

CC @heschik @aclements

[heschi]

dsymutil, I'm pretty sure, surprising nobody.

main.main DIE in go.o, before external linker/dsymutil:

0x00075e8a:     TAG_subprogram [2] *
                 AT_name( "main.main" )
                 AT_low_pc( 0x000000000008f410 )
                 AT_high_pc( 0x000000000008f7f0 )
                 AT_frame_base( call-frame-cfa )
                 AT_decl_file( "/Users/heschi/go/src/hello.go" )
                 AT_external( 0x01 )

and the FDE:

0x00013b3c: FDE
        length: 0x0000002c
   CIE_pointer: 0x00000000
    start_addr: 0x000000000008f410 main.main
    range_size: 0x00000000000003e0 (end_addr = 0x000000000008f7f0)
...

It looks to me like dysmutil just doesn't adjust the addresses in .debug_frame, but I haven't looked at the code to figure out why.

[heschi]

Is this important? I can step through dsymutil and try to figure out what the problem is but this feels like something we'd have to get fixed upstream.

[aarzilli]

Is this important?

No

[erikdw]

@heschik : it seems like this issue is preventing debugging of go plugins on Mac OS X. My project is using go plugins and most of our engineers have Macs, so it's inconvenient to not be able to debug the code.

As I have summarized in a comment in the related delve issue (go-delve/delve#1628), @aarzilli has stated that this particular issue is also a blocker for delve supporting go plugins on Mac. I know very little about how delve, DWARF, etc. work, so I'm not sure if this is the only issue still blocking that from working. However, based on @aarzilli 's statements, I would say the earlier claim that this issue is unimportant is no longer true.

[heschi]

@jeremyfaller I think you would be first stop for this stuff now? Especially if you have a Mac handy -- I let mine rot.

[jeremyfaller]

I'll take a look in the next couple of days. Whatever happens though, this won't make 1.14, so don't expect a mainline fix for quite some time. :)

[jeremyfaller]

Apologies, for the latency. I finally got a chance to take a look at this. As @heschik suspected, it's dsymutil. [un]Surprisingly, dsymutil can't even verify its own output while generating the debug information.

I've filed a bug with apple (7568255), and will follow up here with any responses.

I have little experience debugging dsymutil myself, but I know a few people internally (@google) with some experience. I'll reach out to them next week, and see if they have any insights as well. I'll update this bug as we hear from Apple, or I gain insight from the local knowledgable people.

[aarzilli]

@jeremyfaller did apple ever do anything about this? The problem has just become more pressing because on darwin/arm64 all externally linked executables are PIE (612b119) so on that architecture anything that uses cgo will encounter this problem, see: go-delve/delve#2346.

[jeremyfaller]

Well, I did follow up internally, and hit lots of roadblocks. No one had seen anything, and I just put it on hold mentally until I head from Apple. I never heard, and completely forgot about it. I'll admit it fell off my radar after that though. I'll ping the bug again with Apple, and see if I can pursue some extra channels we have.

[andig]

Any news on this one? Just hit it again and its highly frustrating and confusing when you can't trust your debugger anymore.
Since the delve workaround has been merged I assume I shouldn't even be seeing this with current go and vscode?

[kolkov]

Any news on this one? Just hit it again and its highly frustrating and confusing when you can't trust your debugger anymore.
Since the delve workaround has been merged I assume I shouldn't even be seeing this with current go and vscode?

Yes. The same issue in Golang for me. (

[zonghaishang]

build git:(go-plugin) ✗ go version
go version go1.17.3 darwin/amd64

build git:(go-plugin) ✗ dlv version
Delve Debugger
Version: 1.7.2
Build: $Id: a06f0b8c273325e2160bfd35972c12eda410077c $

// build my plugin.
go build -o codec-xr.so -mod=mod --buildmode=plugin -gcflags "all=-N -l" -ldflags "-compressdwarf=false" github.com/mosn/wasm-sdk/go-plugin/plugins/codecs/xr/main

build output:
https://github.com/zonghaishang/wasm-sdk/tree/go-plugin/go-plugin/build/codecs/xr

➜  build git:(go-plugin) ✗ tree
.
└── codecs
    └── xr
        ├── codec-xr.md5
        ├── codec-xr.so
        ├── egress_xr.json
        ├── ingress_xr.json
        └── mosn_config.json

I can't debug the Go plugin on MAC, I'm going to crash, who has any solution?

[FloThinksPi]

Nop @zonghaishang i saw some people made workarounds that compiled the plugin into the main binary. They made a script that, just for debugging, automatically rearanged the project to compile into a single binary and after compilation restore the initial state. Its a major hack and afaik does not work generically. Yet as i understand the issue is dysmutil and only apple can fix dysmutil.

However i thought the workaround made things better, yet had no time to test it since we ditched plugins due to this issue for now all together.

Just because i assume @jeremyfaller has also other things on his desk ill leave a ping for him here if he heared back 😃
BTW is the bug report to apple public and can be viewed or commented ?

[zonghaishang]

Nop @zonghaishang i saw some people made workarounds that compiled the plugin into the main binary. They made a script that, just for debugging, automatically rearanged the project to compile into a single binary and after compilation restore the initial state. Its a major hack and afaik does not work generically. Yet as i understand the issue is dysmutil and only apple can fix dysmutil.

However i thought the workaround made things better, yet had no time to test it since we ditched plugins due to this issue for now all together.

Just because i assume @jeremyfaller has also other things on his desk ill leave a ping for him here if he heared back 😃 BTW is the bug report to apple public and can be viewed or commented ?

I solved the plug-in debugging problem with container technology.

• build plugin in linux docker: https://github.com/zonghaishang/wasm-sdk/blob/go-plugin/go-plugin/Makefile#L27
• build dlv debug program: https://github.com/zonghaishang/wasm-sdk/blob/go-plugin/go-plugin/build/sidecar/image/Dockerfile
• start debug server: https://github.com/zonghaishang/wasm-sdk/blob/go-plugin/go-plugin/etc/script/run.sh#L18

The screenshot correctly identifies the plug-in information and can also be executed inside the plug-in in a single step(step into :-) .

[kozmod]

👋🏻 @zonghaishang , could you please fix brocken links in your last comment.

[aarzilli]

Change 064f34f cherrypicked from 5231ba2 made this problem start to happen on non-PIE builds on darwin/amd64. I guess what makes this problem happen is not PIE-ness but actually the lack of -pagezero_size being passed to the linker.

I wonder what -pagezero_size does that makes dsymutil work.

cc @cherrymui

[aarzilli]

Crossreferencing https://reviews.llvm.org/D138763

[aarzilli]

Closing, since the bug was in dsymutil and it has been fixed.

[cherrymui]

Thanks for fixing this @aarzilli !