Miscompile on synthesized equatable when compiled with optimizations on arm64(iOS)

[Reflejo]

Describe the bug

We found out that compiling our app with -O and a specific linker arguments ordering breaks synthesized equatable if an xcframework dependency was compile with an older swift version (Xcode 13.4 in this case), on a case like:

// Main App
struct SomeStruct: Equatable {
    let abc: Data
}

func doStuff() {
    let requests: [SomeStruct] = [
        SomeStruct(abc: "a".data(using: .ascii)!)
    ]

    print("First", requests.firstIndex(of: requests[0]))
}

// BrokenLibrary.xcframework
import Foundation

@frozen
public enum ExampleEnum: Equatable {
  case exampleCase(data: Data)
}

In this condition, if the .o files of the main module are passed at the end of the clang ... linking call (as opposed to before the other xcframework arguments), the final linked binary points the data equality of both cases to the same routine
_$s10Foundation4DataV2eeoiySbAC_ACtFZTf4nnd_n [function signature specialization <Arg[2] = Dead> of static Foundation.Data.== infix(Foundation.Data, Foundation.Data) -> Swift.Bool]

Which ends up with the wrong equality result in runtime.

Steps To Reproduce

Steps to reproduce the behavior:

1. Unzip ReproAppClean.zip
2. Open Xcode 14.0, and run the app on an iOS device (arm64)
3. Observe the console
   ...

Expected behavior

The console should print First Optional(0), it prints instead First nil

Environment (please fill out the following information)

• OS: macOS 12.6
• Xcode Version/Tag/Branch (Main App): Version 14.0 (14A309)
• Xcode Version/Tag/Branch (xcframework): Version 13.4 (13F17a)

Additional context

It's important to notice that the LD environment variable is set on Xcode to an ld.py script that invokes clang but with the arguments ordering that breaks this case. It also needs -ObjC (which is set already in the provided Xcode project).

ReproAppClean.zip

[keith]

Adding some more context we discovered here. We think what's happening is the implementation of some function, specifically firstWhere is inlined into the xcframework that was built with an older version of Xcode / stdlib (we haven't checked if the difference is the compiler itself or the SDK). This implementation pulls in a specific == definition, that is somehow incompatible with the definition from the newer compiler / stdlib. Because of this when we call a similar firstWhere function in our code, built with the newer versions, the definition points to the older == definition, which results in the wrong behavior. It seems that the issue is that both == definitions are the same weak symbols, so the first one wins, and in our case the -framework pointing to the xcframework happens to come before the object file from our project, meaning the old symbol wins.

We're generally surprised by the potential for the equatable definition to differ at all, so maybe it's something deeper that changed. There are lots of interesting tidbits about this, for example if you call requests[0] == requests[0] it also returns false, but if you call:

for r in requests {
    print(r == requests[0])
}

It correctly returns true. It seems like somehow these different scenarios of using the equatable definition directly, vs indirectly with firstWhere, forces the correct symbol to be pulled in instead. Specifically when we have _$s10Foundation13__DataStorageC15withUnsafeBytes2in5applyxSnySiG_xSWKXEtKlFSb_Tgq5015$s10Foundation4B26V2eeoiySbAC_ACtFZSbSWXEfU_AA0B0VTf1ncn_nTf4nng_n, all is well.

[keith]

Another thing we were thinking about in general was that if this is a SDK bug, it's possible this wasn't new in Swift itself, but just hadn't surfaced given lower usage of -enable-library-evolution in the community until more recently, and the right situation to actually have an incompatibility

[aschwaighofer]

How did you produce the BrokenLibrary.xcframework library. Can you attach the project to produce it?

[keith]

It's part of the xcodeproj in the zip file, with Xcode 13 selected globally you can run:

xcodebuild archive -scheme BrokenLibrary -archivePath BrokenLibrary-iphoneos.xcarchive -sdk iphoneos SKIP_INSTALL=NO && xcodebuild -create-xcframework -framework BrokenLibrary-iphoneos.xcarchive/Products/Library/Frameworks/BrokenLibrary.framework -output BrokenLibrary.xcframework

[Reflejo]

I'm not sure it makes much of a difference at this point, but a smaller case that also reproduces is if the BrokenLibrary just has

public func equalRoulette() {
    let a: Data = "a".data(using: .ascii)!
    let b: Data = "a".data(using: .ascii)!
    print("Third \(a == b)")
}

(I removed the other type entirely) So any code really that ends up inlining the Data == infix from the (old versioned) library is enough to repro.

To clarify, the print from the library you see here, actually prints the right result (true) but affects the main app the same way I mentioned before

[eeckstein]

This might be a compiler bug in Xcode 13.4, which is fixed in Xcode 14.
It's expected behavior that the linker picks one instance of such a "shared" function, depending on the input file order.

If you still see this problem when building the "BrokenLibrary" with Xcode 14, please reopen this issue (We could not reproduce the issue when building everything with Xcode 14).

[keith]

Thanks for the reply @eeckstein! In that case can is the recommendation to always upgrade the Swift version used to precompile libraries even when using library evolution? What's to stop another bug like this from being introduced in a newer compiler version?

[Reflejo]

@eeckstein Do we know for a fact what bug caused this? I'm curious to understand how this breaks the case more precisely. The library compiled in 13.4 + main app compile in 13.4 works fine, so the bug only manifests when compiling cross library versions. I'm concerned there might be a more fundamental assumptions that break on an optimization pass that might bite us again.

Side note: As you can imagine, people don't have full control about update schedules on the third party libraries they use. I think this sort of bugs break the guarantees of compatibility which is a bit scary. I'm not sure what we should have done differently when deciding to update the toolchain of our app.

[eeckstein]

so the bug only manifests when compiling cross library versions

Okay, that wasn't clear to me.

In this case it's probably not a miscompile of Xcode 13.4. We'll take another look why the version mismatch causes a problem.

[keith]

This bug also reproduces on the arm64 iOS simulator on M1 machines, but doesn't repro on x86_64 simulators. So I guess it's generally arm64 specific, but not iOS platform specific.