cmd/go: install produces executable with invalid signature (macOS)

[koliyo]

I can not launch gopls at all. It is immediately killed

$ gopls
[1]    6414 killed     gopls

I have looked in system console and it seems this is related to code signing:

default	08:59:00.573715+0100	kernel	CODE SIGNING: cs_invalid_page(0x102b24000): p=4159[gopls] final status 0x23020200, denying page sending SIGKILL
default	08:59:00.573730+0100	kernel	CODE SIGNING: process 4159[gopls]: rejecting invalid page at address 0x102b24000 from offset 0x4000 in file "/Users/nils/Work/go/bin/gopls" (cs_mtime:1699430007.975302102 == mtime:1699430007.975302102) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
default	08:59:00.573996+0100	kernel	CODE SIGNING: process 4159[gopls]: rejecting invalid page at address 0x102b24000 from offset 0x4000 in file "/Users/nils/Work/go/bin/gopls" (cs_mtime:1699430007.975302102 == mtime:1699430007.975302102) (signed:1 validated:1 tainted:1 nx:0 wpmapped:0 dirty:0 depth:0)
default	08:59:00.574052+0100	kernel	gopls[4159] Corpse allowed 1 of 5
default	08:59:00.938868+0100	ReportCrash	Formulating fatal 309 report for corpse[4159] gopls
default	08:59:00.940445+0100	osanalyticshelper	creating type 309 as /Users/nils/Library/Logs/DiagnosticReports/.gopls-2023-11-08-085900.ips
default	08:59:00.942224+0100	osanalyticshelper	Saved type '309(<private>)' report (7 of max 25) at /Users/nils/Library/Logs/DiagnosticReports/gopls-2023-11-08-085900.ips
default	08:59:00.942312+0100	osanalyticshelper	xpc log creation type 309 result success: /Users/nils/Library/Logs/DiagnosticReports/gopls-2023-11-08-085900.ips
default	08:59:00.942394+0100	ReportCrash	client log create type 309 result success: /Users/nils/Library/Logs/DiagnosticReports/gopls-2023-11-08-085900.ips
default	08:59:00.942452+0100	ReportCrash	no MetricKit for process gopls type 309 bundleId (null)

Not sure what to do about this

Here is an example of such ips file:

{"app_name":"gopls","timestamp":"2023-11-08 08:59:00.00 +0100","app_version":"","slice_uuid":"3513ca24-39ed-3d73-96b9-9b5acf281169","build_version":"","platform":1,"share_with_app_devs":1,"is_first_party":1,"bug_type":"309","os_version":"macOS 14.1.1 (23B81)","roots_installed":0,"incident_id":"AD4F6056-9001-4D59-9314-23DCE3A4461A","name":"gopls"}
{
  "uptime" : 230,
  "procRole" : "Unspecified",
  "version" : 2,
  "userID" : 501,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro18,2",
  "coalitionID" : 585,
  "osVersion" : {
    "train" : "macOS 14.1.1",
    "build" : "23B81",
    "releaseType" : "User"
  },
  "captureTime" : "2023-11-08 08:59:00.5753 +0100",
  "codeSigningMonitor" : 1,
  "incident" : "AD4F6056-9001-4D59-9314-23DCE3A4461A",
  "pid" : 4159,
  "translated" : false,
  "cpuType" : "ARM-64",
  "roots_installed" : 0,
  "bug_type" : "309",
  "procLaunch" : "2023-11-08 08:59:00.5629 +0100",
  "procStartAbsTime" : 5667297332,
  "procExitAbsTime" : 5667563642,
  "procName" : "gopls",
  "procPath" : "\/Users\/USER\/*\/gopls",
  "parentProc" : "zsh",
  "parentPid" : 3868,
  "coalitionName" : "dev.warp.Warp-Stable",
  "crashReporterKey" : "365F932B-9741-7702-C2F7-EDB5266157E3",
  "responsiblePid" : 536,
  "responsibleProc" : "stable",
  "codeSigningID" : "a.out",
  "codeSigningTeamID" : "",
  "codeSigningFlags" : 587334144,
  "codeSigningValidationCategory" : 0,
  "codeSigningTrustLevel" : 4294967295,
  "instructionByteStream" : {"beforePC":"TtkAlB8gA9XgBwD54wMeqjayAZTgB0D50P\/\/FwAAAAAAAAAAAAAAAA==","atPC":"kAtA+f9jMOvJBQBU\/g8f+P2DH\/j9IwDRgAAAtOEDH6riAx+qCAAAFA=="},
  "sip" : "enabled",
  "vmRegionInfo" : "0x102b26320 is in 0x102b20000-0x103670000;  bytes after start: 25376  bytes before end: 11836639\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  __TEXT                      102b20000-103670000    [ 11.3M] r-x\/r-x SM=COW  ...\/USER\/*\/gopls\n      __DATA_CONST                103670000-1038bc000    [ 2352K] r--\/rw- SM=COW  ...\/USER\/*\/gopls",
  "exception" : {"codes":"0x0000000000000032, 0x0000000102b26320","rawCodes":[50,4340212512],"type":"EXC_BAD_ACCESS","signal":"SIGKILL (Code Signature Invalid)","subtype":"UNKNOWN_0x32 at 0x0000000102b26320"},
  "termination" : {"flags":0,"code":2,"namespace":"CODESIGNING","indicator":"Invalid Page"},
  "ktriageinfo" : "VM - (arg = 0x0) Waiting on busy page was interrupted\nVM - (arg = 0x0) Didn't get back data for this file\nVM - (arg = 0x0) Fault was interrupted\nVM - (arg = 0x0) A memory corruption was found in executable text\n",
  "vmregioninfo" : "0x102b26320 is in 0x102b20000-0x103670000;  bytes after start: 25376  bytes before end: 11836639\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  __TEXT                      102b20000-103670000    [ 11.3M] r-x\/r-x SM=COW  ...\/USER\/*\/gopls\n      __DATA_CONST                103670000-1038bc000    [ 2352K] r--\/rw- SM=COW  ...\/USER\/*\/gopls",
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":4,"task_for_pid":1},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":30274,"threadState":{"x":[{"value":4352059088,"symbolLocation":8744,"symbol":"typerel.*"},{"value":8752},{"value":8752},{"value":4352050336,"symbolLocation":0,"symbol":"runtime.types"},{"value":4354416168},{"value":4353086872,"symbolLocation":1036528,"symbol":"typerel.*"},{"value":4353087072,"symbolLocation":1036728,"symbol":"typerel.*"},{"value":4353086888,"symbolLocation":1036544,"symbol":"typerel.*"},{"value":3},{"value":0},{"value":0},{"value":3},{"value":1374390894608},{"value":3263713395},{"value":37858},{"value":7},{"value":1374390829984},{"value":1374390836384},{"value":0},{"value":1374390836488},{"value":125},{"value":3},{"value":4360589644,"symbolLocation":4,"symbol":"timebase"},{"value":6126694288},{"value":6126694352},{"value":6513804731},{"value":1374390836400},{"value":16},{"value":1374389543328}],"flavor":"ARM_THREAD_STATE64","lr":{"value":4340257956},"cpsr":{"value":536875008},"fp":{"value":1374390836248},"sp":{"value":1374390836256},"esr":{"value":2181038087,"description":"(Instruction Abort) Translation fault"},"pc":{"value":4340212512,"matchesCrashFrame":1},"far":{"value":4340212512}},"queue":"com.apple.main-thread","frames":[{"imageOffset":25376,"symbol":"*\/abi.Name.Name","symbolLocation":0,"imageIndex":0},{"imageOffset":70820,"symbol":"runtime.(*itab).init","symbolLocation":388,"imageIndex":0}]},{"id":30275,"frames":[{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":0},{"value":18446744073709551615},{"value":1},{"value":1},{"value":0},{"value":20000},{"value":10000000},{"value":3},{"value":60},{"value":7961593232,"symbolLocation":0,"symbol":"errno"},{"value":17},{"value":2150627323},{"value":2043},{"value":6135098880},{"value":2152726523},{"value":2043},{"value":6516675220,"symbolLocation":0,"symbol":"__error"},{"value":8112471192},{"value":0},{"value":6135098968},{"value":125},{"value":3},{"value":0},{"value":0},{"value":0},{"value":0},{"value":4354084080,"symbolLocation":1384,"symbol":"go:funcrel.*"},{"value":1},{"value":1374389545408}],"flavor":"ARM_THREAD_STATE64","lr":{"value":4340666120},"cpsr":{"value":1610616832},"fp":{"value":7358459582998995428},"sp":{"value":6135098880},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":4340666120},"far":{"value":0}}},{"id":30276,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":512},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6143519624},{"value":0},{"value":1374389980256},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374389980232},{"value":1374389980296},{"value":6143520992},{"value":0},{"value":0},{"value":512},{"value":513},{"value":768},{"value":18446744073709551520},{"value":1374389546656}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6143519744},"sp":{"value":6143519600},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}},{"id":30277,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":256},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6151957384},{"value":0},{"value":1374390584160},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374390584136},{"value":1374390584200},{"value":6151958752},{"value":0},{"value":0},{"value":256},{"value":257},{"value":512},{"value":18446744073709551520},{"value":1374390591904}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6151957504},"sp":{"value":6151957360},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}},{"id":30278,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":256},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6160411736},{"value":0},{"value":1374391108448},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374391108424},{"value":1374391108488},{"value":6160412896},{"value":0},{"value":0},{"value":256},{"value":257},{"value":512},{"value":18446744073709551520},{"value":1374391116192}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6160411856},"sp":{"value":6160411712},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}},{"id":30279,"frames":[{"imageOffset":20652,"symbol":"__psynch_cvwait","symbolLocation":8,"imageIndex":3},{"imageOffset":30204,"symbol":"_pthread_cond_wait","symbolLocation":1228,"imageIndex":4},{"imageOffset":484712,"symbol":"runtime.pthread_cond_wait_trampoline.abi0","symbolLocation":24,"imageIndex":0},{"imageOffset":478984,"symbol":"runtime.asmcgocall.abi0","symbolLocation":200,"imageIndex":0}],"threadState":{"x":[{"value":260},{"value":0},{"value":0},{"value":0},{"value":0},{"value":160},{"value":0},{"value":0},{"value":6168833288},{"value":0},{"value":1374390585312},{"value":2},{"value":0},{"value":0},{"value":0},{"value":0},{"value":305},{"value":8112471384},{"value":0},{"value":1374390585288},{"value":1374390585352},{"value":6168834272},{"value":0},{"value":0},{"value":0},{"value":1},{"value":256},{"value":18446744073709551584},{"value":1374390592736}],"flavor":"ARM_THREAD_STATE64","lr":{"value":6516938236},"cpsr":{"value":1610616832},"fp":{"value":6168833408},"sp":{"value":6168833264},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6516687020},"far":{"value":0}}}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "arm64",
    "base" : 4340187136,
    "size" : 11862016,
    "uuid" : "3513ca24-39ed-3d73-96b9-9b5acf281169",
    "path" : "\/Users\/USER\/*\/gopls",
    "name" : "gopls"
  },
  {
    "size" : 0,
    "source" : "A",
    "base" : 0,
    "uuid" : "00000000-0000-0000-0000-000000000000"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6513258496,
    "size" : 607000,
    "uuid" : "ec7a3ba0-f9bf-3ab8-a0f4-8622e5606b20",
    "path" : "\/usr\/lib\/dyld",
    "name" : "dyld"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6516666368,
    "size" : 241648,
    "uuid" : "b7751381-1442-30b5-91b9-ad7be461bebe",
    "path" : "\/usr\/lib\/system\/libsystem_kernel.dylib",
    "name" : "libsystem_kernel.dylib"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6516908032,
    "size" : 53236,
    "uuid" : "daf95373-5de6-39a1-a6ce-d87f3f0629cc",
    "path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "name" : "libsystem_pthread.dylib"
  }
],
  "sharedCache" : {
  "base" : 6512508928,
  "size" : 4018896896,
  "uuid" : "80dd42b3-8a52-3caf-9848-54a1c2732864"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=1.0G resident=0K(0%) swapped_out_or_unallocated=1.0G(100%)\nWritable regions: Total=610.0M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=610.0M(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nKernel Alloc Once                   32K        1 \nMALLOC                           522.2M       12 \nMALLOC guard page                   96K        6 \nSTACK GUARD                         80K        5 \nStack                             48.1M        6 \nStack Guard                       56.0M        1 \nVM_ALLOCATE                        1.2G       36 \n__AUTH                             339K       65 \n__AUTH_CONST                      4053K      147 \n__DATA                            2466K      142 \n__DATA_CONST                      11.5M      150 \n__DATA_DIRTY                       363K       58 \n__LINKEDIT                       888.2M        2 \n__OBJC_RO                         70.8M        1 \n__OBJC_RW                         2156K        1 \n__TEXT                           131.1M      156 \ndyld private memory                272K        2 \nshared memory                       16K        1 \n===========                     =======  ======= \nTOTAL                              2.9G      792 \n",
  "legacyInfo" : {
  "threadTriggered" : {
    "queue" : "com.apple.main-thread"
  }
},
  "logWritingSignature" : "8dbd5b1ee96ca3011838b0adc040bf1d106b1d9d",
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "6297d96be2c9387df974efa4",
      "factorPackIds" : {

      },
      "deploymentId" : 240000014
    },
    {
      "rolloutId" : "6112d17137f5d11121dcd4e2",
      "factorPackIds" : {

      },
      "deploymentId" : 240000442
    }
  ],
  "experiments" : [
    {
      "treatmentId" : "a092db1b-c401-44fa-9c54-518b7d69ca61",
      "experimentId" : "64a844035c85000c0f42398a",
      "deploymentId" : 400000019
    }
  ]
}
}

gopls version

Not possible to check, using gopls -v version:

$ gopls -v version
[1]    5720 killed     gopls -v version

But I installed latest gopls version, using VSCode update functionality

go env

What did you do?

What did you expect to see?

What did you see instead?

Editor and settings

Logs

[koliyo]

I was able to work around this issue by resigning the binary:

codesign -f -s - /Users/nils/Work/go/bin/gopls

And now it works:

gopls -v version
Build info
----------
golang.org/x/tools/gopls v0.14.1
    golang.org/x/tools/gopls@v0.14.1 h1:XaTETpi7Q67XO8nftquJitcx+9c2bPclO8Kz2sBVvec=
    github.com/BurntSushi/toml@v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=
    github.com/google/go-cmp@v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
    github.com/sergi/go-diff@v1.1.0 h1:we8PVUC3FE2uYfodKH/nBHMSetSfHDR6scGdBi+erh0=
    golang.org/x/exp/typeparams@v0.0.0-20221212164502-fae10dda9338 h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y=
    golang.org/x/mod@v0.13.0 h1:I/DsJXRlw/8l/0c24sM9yb0T4z9liZTduXvdAWYiysY=
    golang.org/x/sync@v0.4.0 h1:zxkM55ReGkDlKSM+Fu41A+zmbZuaPVbGMzvvdUPznYQ=
    golang.org/x/sys@v0.13.0 h1:Af8nKPmuFypiUBjVoU9V20FiaFXOcuZI21p0ycVYYGE=
    golang.org/x/telemetry@v0.0.0-20231011160506-788d5629a052 h1:1baVNneD/IRxmu8JQdBuki78zUqBtZxq8smZXQj0X2Y=
    golang.org/x/text@v0.13.0 h1:ablQoSUd0tRdKxZewP80B+BaqeKJuVhuRxj/dkrun3k=
    golang.org/x/tools@v0.14.1-0.20231026192422-8b5abd452b28 h1:5YgdZAe2w0x3Xrjv0+GXrI0jvm7qCQK/ySGFfiEHMfU=
    golang.org/x/vuln@v1.0.1 h1:KUas02EjQK5LTuIx1OylBQdKKZ9jeugs+HiqO5HormU=
    honnef.co/go/tools@v0.4.5 h1:YGD4H+SuIOOqsyoLOpZDWcieM28W47/zRO7f+9V3nvo=
    mvdan.cc/gofumpt@v0.4.0 h1:JVf4NN1mIpHogBj7ABpgOyZc65/UUOkKQFkoURsz4MM=
    mvdan.cc/xurls/v2@v2.4.0 h1:tzxjVAj+wSBmDcF6zBB7/myTy3gX9xvi8Tyr28AuQgc=
go: go1.21.3

[findleyr]

I'm unfamiliar with code signing, but found #42684, which also relates to arm64. I don't see how this problem could be specific to gopls: are you able to install and run other go programs using go install? Are you able to run local binaries using go run?

(asking to confirm before I CC compiler folks)

[koliyo]

I have not had any problems with go run, and I tested go install on a local project, and can launch that artifact without problem from the target bin path.

Not sure why I see this problem, I have another dev machine where I have not seen the same issue. Also, not sure I can reproduce the original problem now that I fixed it using codesign explicitly

[koliyo]

Actually, I can reproduce it!

If I delete /Users/nils/Work/go/bin/gopls and then install gopls using Go: Install/Update tools in VSCode I get the original error.

$ ~/Work/go/bin/gopls 
[1]    98838 killed     ~/Work/go/bin/gopls

[findleyr]

Thanks for following up. Can you please try installing gopls@v0.13.2 (an older version) to see if the outcome is different?

go install golang.org/x/tools/gopls@v0.13.2

Then can you try installing another arbitrary binary from x/tools, to see if it works? Say:

go install golang.org/x/tools/cmd/gonew@latest

Can you do gonew --help?

[koliyo]

gopls@v0.13.2
-> Same error as before

gonew@latest
-> Works

gonew --help
usage: gonew srcmod[@version] [dstmod [dir]]
See https://pkg.go.dev/golang.org/x/tools/cmd/gonew.

[findleyr]

Thanks @koliyo for confirming.

CC @golang/compiler -- does anyone have an idea why the gopls binary would require signing? It seems like this just started happening in the past couple weeks, and may be limited to arm64 based on the reports I've seen.

[cherrymui]

macOS on ARM64 requires all binaries to be signed, gopls is not an exception. The go toolchain should already produce a signed binary. What is the output of codesign -vv <binary>? It should say it is valid if it is signed properly.

How is the binary installed? Do you by any chance overwrite an existing binary (like using the cp command to copy to an existing file)? The macOS kernel caches the code signature. If you overwrite an existing binary, the kernel still keeps the old signature cached, and when the binary is launched it sees a mismatch and kills the program. Deleting the old binary and creating a new one would fix. Rebooting the machine would also clear the cache. I think the go command always arranges it to create a new binary, and the go linker also calls mmap to invalidate the kernel cache. But if you copy the file to an existing file after build, that would not help.

Do you just run the go install command and run the binary right after? Or there is some other step? Thanks.

[koliyo]

codesign -vv ~/Work/go/bin/gopls
/Users/nils/Work/go/bin/gopls: invalid signature (code or signature have been modified)
In architecture: arm64

This is a "fresh" binary, installed with go install golang.org/x/tools/gopls@latest

I do not do any other operation than the install command. Normally I install gotools using VSCode.

It is possible that I have used gopls from homebrew at some point in the past, not sure, but that was not recently, and should be another install directory

[findleyr]

Hmm. I'm a bit out of my depth here, but here are a couple ideas to try based on @cherrymui's comment:

• Can you try deleting the gopls binary before installing? (you may have already tried this).
• Can you try installing into a different directory?

TEMP_GOBIN=$(mktemp -d)
GOBIN=$TEMP_GOBIN go install golang.org/x/tools/gopls@latest
$TEMP_GOBIN/gopls version