Segmentation fault on a specific machine when using a binary built with the static Linux SDK

[Cyberbeni]

Description

I have 2 machines. An Intel J4105 based one running Ubuntu 20.04 which doesn't have this issue and a Ryzen 1700x based one running Nobara 40, which does have this issue.

The crash happens using the binary from the artifactbundle found here: https://github.com/nicklockwood/SwiftFormat/releases/tag/0.55.2

Workspace I'm using for testing: Cyberbeni/Wiring@4f79227

When I run swiftformat, I get a segmentation fault:

> ~/Downloads/swiftformat_linux .
Running SwiftFormat...
Reading config file at /home/cyberbeni/Developer/Wiring/.swiftformat
[1]    6165 segmentation fault (core dumped)  ~/Downloads/swiftformat_linux .

But when I run this program in verbose mode, I don't get a crash:

> ~/Downloads/swiftformat_linux --verbose .
Running SwiftFormat...
Reading config file at /home/cyberbeni/Developer/Wiring/.swiftformat
...
SwiftFormat completed in 0.16s.
0/31 files formatted.

Same behaviour when running the same binary with podman: ghcr.io/nicklockwood/swiftformat:0.55.2

I also don't get a crash when running the code in the linked workspace inside a podman container.

Also no crash when using the container of version 0.54.5, which was built without the static SDK on swift:focal and copied over to swift:focal-slim

Also no crash when building locally without the static SDK with the following command: swift build --product swiftformat --configuration release -Xlinker -s

cc nicklockwood/SwiftFormat#1930

Reproduction

See description.

Stack dump

> coredumpctl info 
           PID: 4738 (swiftformat_lin)
           UID: 1000 (cyberbeni)
           GID: 1001 (cyberbeni)
        Signal: 11 (SEGV)
     Timestamp: Tue 2024-11-26 14:58:19 CET (9min ago)
  Command Line: /home/cyberbeni/Downloads/swiftformat_linux .
    Executable: /home/cyberbeni/Downloads/swiftformat_linux
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole@27e0da96d4f6401bba8ab0766af41051.service
          Unit: user@1000.service
     User Unit: app-org.kde.konsole@27e0da96d4f6401bba8ab0766af41051.service
         Slice: user-1000.slice
     Owner UID: 1000 (cyberbeni)
       Boot ID: ca63284dfda64320ae40facc142760fe
    Machine ID: 02f7a45841774093b59a960034641923
      Hostname: nobara-pc
       Storage: /var/lib/systemd/coredump/core.swiftformat_lin.1000.ca63284dfda64320ae40facc142760fe.4738.1732629499000000.zst (present)
  Size on Disk: 12.9M
       Message: Process 4738 (swiftformat_lin) of user 1000 dumped core.
                
                Module /home/cyberbeni/Downloads/swiftformat_linux without build-id.
                Stack trace of thread 4743:
                #0  0x00000000017cd57c n/a (/home/cyberbeni/Downloads/swiftformat_linux + 0x15cd57c)
                ELF object binary architecture: AMD x86-64

From /var/lib/systemd/coredump:

coredump.zip

Expected behavior

Swift files are formatted inside the folder that is passed in as an argument.

Environment

FROM --platform=$BUILDPLATFORM swift:6.0.1 AS builder
WORKDIR /workspace
RUN swift sdk install \
	https://download.swift.org/swift-6.0.1-release/static-sdk/swift-6.0.1-RELEASE/swift-6.0.1-RELEASE_static-linux-0.0.1.artifactbundle.tar.gz \
	--checksum d4f46ba40e11e697387468e18987ee622908bc350310d8af54eb5e17c2ff5481

Also tried with 6.0.2:

FROM --platform=$BUILDPLATFORM swift:6.0.2 AS build
WORKDIR /workspace
RUN swift sdk install \
	https://download.swift.org/swift-6.0.2-release/static-sdk/swift-6.0.2-RELEASE/swift-6.0.2-RELEASE_static-linux-0.0.1.artifactbundle.tar.gz \
	--checksum aa5515476a403797223fc2aad4ca0c3bf83995d5427fb297cab1d93c68cee075

Additional information

No response

[Cyberbeni]

Played around with some different arguments, got a file/line for this crash:

> ~/Downloads/swiftformat_linux --rules unusedArguments .
Running SwiftFormat...
Swift/NativeSet.swift:736: Fatal error
[1]    13086 illegal hardware instruction (core dumped)  ~/Downloads/swiftformat_linux --rules unusedArguments .
> coredumpctl info
           PID: 13086 (swiftformat_lin)
           UID: 1000 (cyberbeni)
           GID: 1001 (cyberbeni)
        Signal: 4 (ILL)
     Timestamp: Tue 2024-11-26 18:01:30 CET (9s ago)
  Command Line: /home/cyberbeni/Downloads/swiftformat_linux --rules unusedArguments .
    Executable: /home/cyberbeni/Downloads/swiftformat_linux
 Control Group: /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.kde.konsole@1c9c4c5ccb874af0b>
          Unit: user@1000.service
     User Unit: app-org.kde.konsole@1c9c4c5ccb874af0be2965e0bcdb4108.service
         Slice: user-1000.slice
     Owner UID: 1000 (cyberbeni)
       Boot ID: 915a064ecd26443dad3a28cdb0dc8dc2
    Machine ID: 02f7a45841774093b59a960034641923
      Hostname: nobara-pc
       Storage: /var/lib/systemd/coredump/core.swiftformat_lin.1000.915a064ecd26443dad3a28cdb0dc8dc2.13086.17>
  Size on Disk: 12.9M
       Message: Process 13086 (swiftformat_lin) of user 1000 dumped core.
                
                Module /home/cyberbeni/Downloads/swiftformat_linux without build-id.
                Stack trace of thread 13087:
                #0  0x0000000000d19e61 n/a (/home/cyberbeni/Downloads/swiftformat_linux + 0xb19e61)
                ELF object binary architecture: AMD x86-64

Same command with the locally built version with the non-static SDK runs just fine.

[al45tair]

My guess is that there's some issue with swift-format where it behaves differently with Musl versus Glibc, and in the former case triggers an assertion failure or similar in the Set code somehow. It's probably an issue with swift-format itself making some assumption or other, at a guess. It might help if you installed the backtracer on the system where it's crashing, then you get an actual crash backtrace. Also, if you can reproduce this in a container using e.g. Docker or Podman, and post instructions for doing that here, that would be very helpful in investigating further.

[Cyberbeni]

Also, if you can reproduce this in a container using e.g. Docker or Podman, and post instructions for doing that here, that would be very helpful in investigating further.

If run inside a container, it only crashes about 70% of the time where running on the host machine crashes every time. Also if I run on the machine with Intel J4105, it doesn't crash either way, only on the machine with AMD 1700x. So I don't think adding the complexity of a container would help.

I have now built it locally in debug mode without stripping, instead of inside a docker container in release mode with stripping, the second crash now produces a stack trace with more than one address (I'm using Swift 6.0.1):

> swift build --product swiftformat --swift-sdk x86_64-swift-linux-musl
...
> ../SwiftFormat/.build/debug/swiftformat --rules unusedArguments .
Running SwiftFormat...
Swift/NativeSet.swift:736: Fatal error
Current stack trace:
0    <unknown>                          0x00000000012e9581
1    <unknown>                          0x0000000000efa40d
2    <unknown>                          0x0000000000efa1cd
3    <unknown>                          0x0000000000efa08e
4    <unknown>                          0x00000000010b9aad
5    <unknown>                          0x00000000010b9cce
6    <unknown>                          0x0000000000f88b0a
7    <unknown>                          0x0000000000f883de
8    <unknown>                          0x000000000117a9cd
9    <unknown>                          0x00000000007fc516
10   <unknown>                          0x00000000007fa989
11   <unknown>                          0x0000000000a6ccf8
12   <unknown>                          0x0000000000a80545
13   <unknown>                          0x0000000000a6de32
14   <unknown>                          0x0000000000a6fd5e
15   <unknown>                          0x00000000008222b8
16   <unknown>                          0x000000000139bdb7
17   <unknown>                          0x000000000139a3f4
[1]    57262 illegal hardware instruction (core dumped)  ../SwiftFormat/.build/debug/swiftformat --rules unusedArguments .

As for the first crash, it might be some race condition issue in SwiftFormat? We are using a concurrent thread when not in verbose mode. But turning on Thread Sanitizer gives no errors on macOS or with Glibc.

let concurrent = !verbose
let completionQueue = DispatchQueue(label: "swiftformat.enumeration")
let queue = concurrent ? DispatchQueue.global(qos: .userInitiated) : completionQueue

It might help if you installed the backtracer on the system where it's crashing

What do I need to install? (I'm on Nobara 40, which is based on Fedora 40) If I build with Glibc and intentially put in a fatalError, it correctly produces a stack trace with symbol names:

> ../SwiftFormat/.build/debug/swiftformat --rules unusedArguments .
CommandLineTool/main.swift:64: Fatal error: HERE

💣 Program crashed: Illegal instruction at 0x00007f0101ce4a00

Thread 0 "swiftformat" crashed:

0 0x00007f0101ce4a00 _assertionFailure(_:_:file:line:flags:) + 496 in libswiftCore.so
1 foo() + 98 in swiftformat at /home/cyberbeni/Developer/SwiftFormat/CommandLineTool/main.swift:64:5

    62│ 
    63│ func foo() {
    64│     fatalError("HERE")                                                                                                                     
      │     ▲
    65│ }
    66│

2 CommandLineTool_main + 17 in swiftformat at /home/cyberbeni/Developer/SwiftFormat/CommandLineTool/main.swift:67:1

    65│ }
    66│ 
    67│ foo()                                                                                                                                      
      │ ▲
    68│ 
    69│ private var stderr = FileHandleOutputStream(filehandle: .standardError)

3 0x00007f0100f55088 __libc_start_call_main + 119 in libc.so.6
4 0x00007f0100f5514b __libc_start_main@GLIBC_2.2.5 + 138 in libc.so.6

Backtrace took 0.17s

[1]    6436 illegal hardware instruction (core dumped)  ../SwiftFormat/.build/debug/swiftformat --rules unusedArguments .

[Cyberbeni]

I switched to a Ryzen 3700x, both issues still present.

[al45tair]

It might help if you installed the backtracer on the system where it's crashing
What do I need to install? (I'm on Nobara 40, which is based on Fedora 40)

If you look in the SDK bundle, in musl-<version>.sdk/<architecture>/usr/libexec/swift/linux-static you will find a swift-backtrace-static executable. If you copy that into the same directory as your program and rename it to swift-backtrace, I think the runtime will find it and you should get a backtrace.

[Cyberbeni]

> ../SwiftFormat/.build/debug/swiftformat .
Running SwiftFormat...
Reading config file at /home/cyberbeni/Developer/Wiring/.swiftformat

💣 Program crashed: Signal 11: Backtracing from 0x19c670c...swift-backtrace: unable to fetch crash info.
 failed

[1]    8667 segmentation fault (core dumped)  ../SwiftFormat/.build/debug/swiftformat .

> ../SwiftFormat/.build/debug/swiftformat --rules unusedArguments .
Running SwiftFormat...
Swift/NativeSet.swift:736: Fatal error

💣 Program crashed: Signal 4: Backtracing from 0xefaea1...swift-backtrace: unable to fetch crash info.
 failed

[1]    8392 illegal hardware instruction (core dumped)  ../SwiftFormat/.build/debug/swiftformat --rules unusedArguments .

[al45tair]

unable to fetch crash info.

:-( That's quite an unusual error. I wouldn't normally expect to see that message. Is this system locked-down in some way? Or is one of these programs running with unusual permissions somehow?

[Cyberbeni]

No unusual permissions, just the standard next-next-finish installation.
Same error on Ubuntu 24.04.1 with Swift 6.0.3

[Cyberbeni]

When doing to same with an empty project containing just a fatalError, it hangs at 💣 Program crashed: Signal 4: Backtracing from {address}...
If I don't move the backtracer next to my binary, then I get a stack trace with a couple <unknown>s

[Cyberbeni]

Seems like at least one other person had issues with the backtracer when using the static SDK: https://forums.swift.org/t/unable-to-generate-a-backtrace-using-swift-6-static-linux-sdk/74215

[Cyberbeni]

Tried using lldb (Swift 6.0.3, swift-backtrace from the static sdk is next to the binary):

> ../swift/6.0.3/bin/lldb .build/debug/swiftformat
(lldb) target create ".build/debug/swiftformat"
Current executable set to '/home/cyberbeni/Developer/SwiftFormat/.build/debug/swiftformat' (x86_64).
(lldb) process launch -- .
Process 5488 launched: '/home/cyberbeni/Developer/SwiftFormat/.build/debug/swiftformat' (x86_64)
Running SwiftFormat...
Reading config file at /home/cyberbeni/Developer/SwiftFormat/.swiftformat
Process 5488 stopped
* thread #67, name = 'swiftformat', stop reason = signal SIGSEGV: invalid permissions for mapped object (fault address: 0x7ffff5d44f50)
    frame #0: 0x0000000000e8fa97 swiftformat`swift::runtime::AccessSet::insert(this=0x00007ffff5d65a90, access=0x0000000000000000, pc=0x000000000088aed8, pointer=0x00007ffff7c222e0, flags=Tracking) at Exclusivity.cpp:157
Target 0: (swiftformat) stopped.
(lldb) bt all
LLDB diagnostics will be written to /tmp/diagnostics-346d34
Please include the directory content when filing a bug report
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: ../swift/6.0.3/bin/lldb .build/debug/swiftformat
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  lldb          0x00005d1b3e3cff48
1  lldb          0x00005d1b3e3ce2de
2  lldb          0x00005d1b3e3d0888
3  libc.so.6     0x00007567ae245320
4  liblldb.so.17 0x00007567afa344e1
5  liblldb.so.17 0x00007567afa33d05
6  liblldb.so.17 0x00007567afa33186
7  liblldb.so.17 0x00007567afa32913
8  liblldb.so.17 0x00007567b1754ab9
9  liblldb.so.17 0x00007567b1750808
10 liblldb.so.17 0x00007567afa31f34
11 liblldb.so.17 0x00007567b0290c52
12 liblldb.so.17 0x00007567b0292738
13 liblldb.so.17 0x00007567b029310b
14 liblldb.so.17 0x00007567b0293237
15 liblldb.so.17 0x00007567b0292371
16 liblldb.so.17 0x00007567b0282a5a
17 liblldb.so.17 0x00007567b027fde9
18 liblldb.so.17 0x00007567afa301f6
19 liblldb.so.17 0x00007567af9a7fdf
20 liblldb.so.17 0x00007567af9afeb5
21 liblldb.so.17 0x00007567af99fd26
22 liblldb.so.17 0x00007567afc3d323
23 liblldb.so.17 0x00007567afc3cd27
24 liblldb.so.17 0x00007567af68582a
25 liblldb.so.17 0x00007567af685896
26 liblldb.so.17 0x00007567af5caeaa
27 liblldb.so.17 0x00007567af5aed27
28 liblldb.so.17 0x00007567af5b1acd
29 liblldb.so.17 0x00007567af5d926c
30 liblldb.so.17 0x00007567af933623
31 liblldb.so.17 0x00007567af5736d6
32 liblldb.so.17 0x00007567af57299a
33 liblldb.so.17 0x00007567af57299a
34 liblldb.so.17 0x00007567af572917
35 liblldb.so.17 0x00007567af721c68
36 liblldb.so.17 0x00007567af721e07
37 liblldb.so.17 0x00007567af722216
38 liblldb.so.17 0x00007567af7283f7
39 liblldb.so.17 0x00007567af77242e
40 liblldb.so.17 0x00007567afe4cf26
41 liblldb.so.17 0x00007567afe56750
42 liblldb.so.17 0x00007567af64e81b
43 liblldb.so.17 0x00007567af6431a1
44 liblldb.so.17 0x00007567afe1bb45
45 liblldb.so.17 0x00007567af64eba1
46 liblldb.so.17 0x00007567af6431a1
47 liblldb.so.17 0x00007567af646ebe
48 liblldb.so.17 0x00007567af57cdca
49 liblldb.so.17 0x00007567af55bc26
50 liblldb.so.17 0x00007567af648d68
51 liblldb.so.17 0x00007567af37089c lldb::SBDebugger::RunCommandInterpreter(bool, bool) + 156
52 lldb          0x00005d1b3e3b7b3e
53 lldb          0x00005d1b3e3b87a8
54 libc.so.6     0x00007567ae22a1ca
55 libc.so.6     0x00007567ae22a28b __libc_start_main + 139
56 lldb          0x00005d1b3e3b3e95

💣 Program crashed: Bad pointer dereference at 0x000000000001f927

Thread 0 "lldb" crashed:

0 0x00007567afa344e1 std::optional<swift::remote::RemoteAbsolutePointer> swift::remote::MetadataReader<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > >, swift::reflection::TypeRefBuilder>::resolveRelativeIndirectableField<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const, swift::RelativeIndirectablePointer<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const, true, int, swift::RuntimeTarget<8u>::StoredSignedPointer> >(swift::remote::RemoteRef<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const>, swift::RelativeIndirectablePointer<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const, true, int, swift::RuntimeTarget<8u>::StoredSignedPointer> const&) + 17 in liblldb.so.17.0.0

Backtrace took 2.01s

From another shell, please run

lldb --attach-pid 5433 -o c
Press any key once LLDB is attached, or A to abort

For the other case, lldb crashes even before me trying to get bt all:

> ../swift/6.0.3/bin/lldb .build/debug/swiftformat
(lldb) target create ".build/debug/swiftformat"
Current executable set to '/home/cyberbeni/Developer/SwiftFormat/.build/debug/swiftformat' (x86_64).
(lldb) process launch -- --rules unusedArguments .
Process 5721 launched: '/home/cyberbeni/Developer/SwiftFormat/.build/debug/swiftformat' (x86_64)
Running SwiftFormat...
Swift/NativeSet.swift:736: Fatal error
LLDB diagnostics will be written to /tmp/diagnostics-275c03
Please include the directory content when filing a bug report
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  lldb          0x00006404fc007f48
1  lldb          0x00006404fc0062de
2  lldb          0x00006404fc008888
3  libc.so.6     0x00007c4470e45320
4  liblldb.so.17 0x00007c44726344e1
5  liblldb.so.17 0x00007c4472633d05
6  liblldb.so.17 0x00007c4472633186
7  liblldb.so.17 0x00007c4472632913
8  liblldb.so.17 0x00007c4474354ab9
9  liblldb.so.17 0x00007c4474350808
10 liblldb.so.17 0x00007c4472631f34
11 liblldb.so.17 0x00007c4472e90c52
12 liblldb.so.17 0x00007c4472e92738
13 liblldb.so.17 0x00007c4472e9310b
14 liblldb.so.17 0x00007c4472e93237
15 liblldb.so.17 0x00007c4472e92371
16 liblldb.so.17 0x00007c4472e82a5a
17 liblldb.so.17 0x00007c4472e7fde9
18 liblldb.so.17 0x00007c44726301f6
19 liblldb.so.17 0x00007c44725a7fdf
20 liblldb.so.17 0x00007c44725afeb5
21 liblldb.so.17 0x00007c447259fd26
22 liblldb.so.17 0x00007c447283d323
23 liblldb.so.17 0x00007c447283cd27
24 liblldb.so.17 0x00007c447228582a
25 liblldb.so.17 0x00007c4472285896
26 liblldb.so.17 0x00007c44721caeaa
27 liblldb.so.17 0x00007c44721aed27
28 liblldb.so.17 0x00007c44721ae117
29 liblldb.so.17 0x00007c44721c3312
30 liblldb.so.17 0x00007c44721b8298
31 liblldb.so.17 0x00007c44721b82ed
32 liblldb.so.17 0x00007c44721bada6
33 liblldb.so.17 0x00007c4472533441
34 liblldb.so.17 0x00007c44721736d6
35 liblldb.so.17 0x00007c447217299a
36 liblldb.so.17 0x00007c447217299a
37 liblldb.so.17 0x00007c4472172917
38 liblldb.so.17 0x00007c4472321c68
39 liblldb.so.17 0x00007c4472321e07
40 liblldb.so.17 0x00007c4472322216
41 liblldb.so.17 0x00007c44723283f7
42 liblldb.so.17 0x00007c447237242e
43 liblldb.so.17 0x00007c44722f6db9
44 liblldb.so.17 0x00007c44722f5d0a
45 liblldb.so.17 0x00007c4472160810
46 liblldb.so.17 0x00007c447216111b
47 liblldb.so.17 0x00007c44721640a9
48 liblldb.so.17 0x00007c447222abe9
49 libc.so.6     0x00007c4470e9ca94
50 libc.so.6     0x00007c4470f29c3c

💣 Program crashed: Bad pointer dereference at 0x0000000000000000

Thread 18 "dbg.evt-handler" crashed:

0 0x00007c44726344e1 std::optional<swift::remote::RemoteAbsolutePointer> swift::remote::MetadataReader<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > >, swift::reflection::TypeRefBuilder>::resolveRelativeIndirectableField<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const, swift::RelativeIndirectablePointer<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const, true, int, swift::RuntimeTarget<8u>::StoredSignedPointer> >(swift::remote::RemoteRef<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const>, swift::RelativeIndirectablePointer<swift::TargetContextDescriptor<swift::External<swift::NoObjCInterop<swift::RuntimeTarget<8u> > > > const, true, int, swift::RuntimeTarget<8u>::StoredSignedPointer> const&) + 17 in liblldb.so.17.0.0

Backtrace took 2.16s

From another shell, please run

lldb --attach-pid 5639 -o c
Press any key once LLDB is attached, or A to abort