runtime/pprof: Go build ID is not recorded in pprof profiles, only GNU build ID

[aalexand]

Go version

1.22.5

Output of go env in your module/workspace:

$ go env
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/aalexand/.cache/go-build'
GOENV='/home/aalexand/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/aalexand/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/aalexand/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/aalexand/.gimme/versions/go1.22.5.linux.amd64'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/aalexand/.gimme/versions/go1.22.5.linux.amd64/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22.5'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4077799602=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I'm using this self-profiling toy program:

$ cat main.go
package main

import (
	"log"
	"os"
	"runtime/pprof"
)

func main() {
	f, err := os.Create("profile.pb.gz")
	if err != nil {
		log.Fatal(err)
	}
	pprof.StartCPUProfile(f)
	defer pprof.StopCPUProfile()
	var i int64
	for i = 0; i < (1 << 33); i++ {
	}
}

When I build it with go build main.go, I can see that, as expected by default, this pure Go binary only has the Go build ID and not the GNU build ID:

$ file main
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=llrn1go725_F2vCvvETz/OITeRu6kDScHG6FVjdK8/R7ABD8hl4lkeyDfz55iM/uoTostDrfB5kdwhy6UpG, with debug_info, not stripped

$ readelf -n main

Displaying notes found in: .note.go.buildid
  Owner                Data size 	Description
  Go                   0x00000053	GO BUILDID
   description data: 6c 6c 72 6e 31 67 6f 37 32 35 5f 46 32 76 43 76 76 45 54 7a 2f 4f 49 54 65 52 75 36 6b 44 53 63 48 47 36 46 56 6a 64 4b 38 2f 52 37 41 42 44 38 68 6c 34 6c 6b 65 79 44 66 7a 35 35 69 4d 2f 75 6f 54 6f 73 74 44 72 66 42 35 6b 64 77 68 79 36 55 70 47

Unexpectedly, when I run this program, the recorded profile does not capture the build ID:

$ ./main

$ pprof -raw profile.pb.gz | grep -A10 Mappings
Mappings
1: 0x400000/0x4ac000/0x0 /tmp/main  [FN]

If I ask Go linker to generate the GNU build ID for the binary using the -B gobuildid linker flag added in #61469, then the profile records the build ID as expected:

$ go build -ldflags "-B gobuildid" main.go

$ file main
main: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=f4b5d514bc46fad9417898216b23910ae874a85d, with debug_info, not stripped

$ readelf -n main

Displaying notes found in: .note.gnu.build-id
  Owner                Data size 	Description
  GNU                  0x00000014	NT_GNU_BUILD_ID (unique build ID bitstring)
    Build ID: f4b5d514bc46fad9417898216b23910ae874a85d

Displaying notes found in: .note.go.buildid
  Owner                Data size 	Description
  Go                   0x00000053	GO BUILDID
   description data: 45 72 5a 36 6f 30 30 37 79 53 35 48 4c 67 41 7a 51 66 6e 52 2f 42 5a 53 51 58 54 4b 49 35 53 61 61 4f 4d 6e 65 49 36 63 56 2f 52 37 41 42 44 38 68 6c 34 6c 6b 65 79 44 66 7a 35 35 69 4d 2f 73 58 6a 56 4b 38 6d 52 58 79 35 4d 79 41 73 46 46 52 6d 74

$ ./main

$ pprof -raw profile.pb.gz | grep -A10 Mappings
Mappings
1: 0x400000/0x4ac000/0x0 /tmp/main f4b5d514bc46fad9417898216b23910ae874a85d [FN]

What did you see happen?

See above.

What did you expect to see?

This happens because src/runtime/pprof/elf.go only reads the GNU build ID, not the Go build ID. I would expect instead to see runtime/pprof record the Go build ID when it's present and fall back to the GNU build ID when the Go build ID is missing. This would be similar to the behavior of the fileutility.

A related issue is that as far as I know Linux perf has its own logic for extracting the build ID as well and it does not record the Go build ID either. I think we in general should be careful with introducing private extensions to recording build ID in the binary since this may have cascade effects on the compatibility of existing profiling and debugging toolchains. It would be good if everything would just record the build ID as the standard GNU build ID.

[gabyhelp]

Related Issues and Documentation

• runtime/pprof: system information in output #59307
• runtime/pprof: Include build id in generated pprof profiles #62091 (closed)
• runtime/pprof: incorrect function names for generics functions #64528 (closed)
• cmd/go: go buildid actionID part is not reproducible with clean cache #59553 (closed)
• runtime/pprof: include build ID in generated profile protos #19431 (closed)
• cmd/go: build ID depends on the full path of Go files passed on the command line #48557 (closed)
• runtime/pprof: spotty tests for several builders #8949 (closed)
• cmd/go: value of -buildmode in buildinfo can differ from what is used in build ID #63559
• runtime/pprof: collecting a CPU profile takes ~200ms of wall time #63043
• cmd/pprof: panics if profile is not found #67626 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[prattmic]

I agree, we should be including a build-id in the profile for pure Go binaries.

I would expect instead to see runtime/pprof record the Go build ID when it's present and fall back to the GNU build ID when the Go build ID is missing.

I wonder if this would break any tools that expect a GNU build ID? Most cgo programs should have a GNU build ID, so the build ID present in profiles would change from those. That said, I'm not sure if any tools actually look at build ID in profiles.

From your last paragraph, am I understanding correctly that you'd prefer we always generate a GNU build id, and use that instead of the Go build ID? We got close with https://go.dev/cl/511475, but IIRC there was some issue with convergence in toolchain binaries that prevented it from being default (cc @cherrymui may recall better?).

[aalexand]

From your last paragraph, am I understanding correctly that you'd prefer we always generate a GNU build id, and use that instead of the Go build ID?

I think that's what I would prefer. Otherwise this becomes a (number of profiling / debugging tools) x (number of languages) problem. For example, we know now that file tool added support for Go build ID at some point. But runtime/pprof, Linux perf and pprof CLI are not aware of it. We can add it there and ensure that file, Linux perf and pprof CLI use the same prioritization to disambiguate when necessary (use Go build ID, fall back to GNU build ID) but there are most likely more tools to update, plus what if tomorrow Rust / Ruby / etc decides to add its own build ID note too?

An alternative would be to teach all profilers and debuggers to compute their own build ID to become independent of any ELF notes, but that is also nuanced - e.g. Linux perf in system-wide mode can observe events in hundreds of files, computing a custom hash of a binary will have to either be partial (e.g. first N bytes of the binary) or have potentially high overhead since a full scan of hundreds of binaries can put some pressure on the system. Also, a custom build ID wouldn't be compatible with build ID in debuginfo packages which is often critical to symbolize system libraries, so a fallback to GNU build ID would have to exist in most profilers anyway.

[mknyszek]

CC @golang/runtime

[cherrymui]

#63934 is related. We currently don't generate GNU build ID by default. If we fix #63934 (we have a plan for that) and enable GNU build ID by default, it will automatically included in the profile.

[aalexand]

and enable GNU build ID by default

That would be great. But even then I wonder whether runtime/pprof should still support the Go build ID since someone coudl turn off the GNU build ID generation. Though the priority of that support would be much lower than now. Personally I would prefer if Go would only support GNU build ID but I don't know why the Go build ID was added in the first place. Plus taking it away is probably impossible in practice now since users may depend on it.

[cherrymui]

The Go build ID is introduced for the Go toolchain's own build cache and reproducibility. It wants a portable format that is not tied to ELF. And it also has multiple components, like an action ID and a content ID, and some flexibility as the Go toolchain evolves. So using ELF build ID for that purpose would be difficult. The Go build ID is not intended to be used by anything outside of the Go toolchain.

Yeah, one could turn GNU build ID off, or overwrite it. Technically one could also turn off the Go build ID (the linker's -buildid flag). That would result in a binary that is not cacheable, and perhaps not being analyzable by some Go tools, but it is still a runnable executable. That would be rare.

[aalexand]

I see. If the Go build ID is not intended to be used outside the Go toolchain then getting the GNU build ID enabled by default seems like the right path forward. And perhaps when that happens, this issue can simply be closed as WAI.