Sporadic build failures caused by native linking error

[jwholme2]

Steps to Reproduce

My iOS builds in AppCenter fail if

1. Xamarin iOS 12.2 and Mono 5.16 is selected

The builds did work for a few days if I select Mono 5.12, but now they are failing again. It's always a native linker issue.

Expected Behavior

iOS builds work.

Actual Behavior

iOS builds fail.

Environment

AppCenter builds.

Build Logs

https://gist.github.com/jwholme2/af89f2979e45289f2d0fa7b034e68050

Example Project (If Possible)

[spouliot]

mtouch is shipped with Xamarin.iOS and follows the same version convention (e.g. 5.x was released 7 years ago). What you are mentioning looks like mono versions - which are not related to the SDK (beside providing csc and msbuild).

The partial log linked shows native linking errors - which are unrelated to the managed linker setting (e.g. Link SDK).

There must be other things different between the builds but we do not have the details to help you.

Please include the missing informations (all versions) and the full build log so we can investigate this further. Thanks!

[jwholme2]

Thank you. Yes - I had confused mtouch and Mono. I had the builds working for 1 day when I selected Xamarin.iOS 12.2/Mono 5.12 in AppCenter, but today they have started failing again (in AppCenter). I was able to reproduce the issue locally before upgrading to the latest Xcode version. The latest Xcode gives another issue, but again references mtouch.

What's the best way to debug the native linking errors? I probably won't be able to post the build logs here....

Thanks!

[spouliot]

Just changing mono versions should not influence native linking - so it's not a common/known error.

The errors are reported at the end of the build logs (with MTxxxx errors), however what caused them is shown earlier in the logs (when the Apple SDK tool is used). That why we need to see the logs to investigate. You can do the same (read the logs) but, beside looking for the same error earlier, there's step-by-step instructions on doing this (it really depends on what you find at each step until the root cause is found).

[jwholme2]

I can send some snippets of the build log here....is there a way to send the build logs to you privately ?

[jwholme2]

I've uploaded the Xamarin.iOS build log from AppCenter here:

https://gist.github.com/jwholme2/8a12eb9ceb87c897da0e49a587187f09

[jwholme2]

This seemed suspicious, but it is present in our passing build logs too:

https://gist.github.com/jwholme2/8a12eb9ceb87c897da0e49a587187f09#file-gistfile1-txt-L274

[spouliot]

it's hard to say, the verbosity level is minimal, a build with "-v -v -v -v" in the Additional mtouch arguments (of the project options) would give more details.

There's a size limit for 32bits (armv7) binaries in Apple's native linker. You could be hitting this...

What happens if you try to build only for arm64 ?

Also check if you have the managed linker enabled in your build. It reduce the app size a lot, so if you disable it that makes it more likely you'll go over the armv7 limit.

[spouliot]

�Also the los shows Mono 5.12 and Xamarin.iOS 12.2.1.11 are being used

/Library/Frameworks/Mono.framework/Versions/5.12.0/
/Library/Frameworks/Xamarin.iOS.framework/Versions/12.2.1.11/

so I'm not sure where that 5.16 relates to ?!? there's no mention of 5.16 anywhere in your logs

[jwholme2]

Thank you! I will look into the size limits causing our sporadic build failures.

Re: the mono version: Changing it to 5.12 seemed to help get our builds green again for ~2 days, but they started going red again last night. So it may be a bit of a red herring. I updated the issue description and title to reflect this.

[spouliot]

ok, then it really sound like the old 32bits native linking issue :(

sadly the error it gives are never the same (across different Xcode releases) so it's not easy to diagnose (and not possible to have a nicer, accurate error message)

you might be close to that size limit so would explain why small changes (in versions or in your code) will trigger this randomly

[jwholme2]

What happens if you try to build only for arm64 ?

This builds just fine.

[jwholme2]

I guess the only option is to only support arm64 ?

[spouliot]

I guess the only option is to only support arm64 ?

That would be a drastic last resort. It's quite uncommon to have such large release binaries unless you:

• have disabled the managed linker [1]; or

• are using a lot of native libraries (e.g. bindings) and, because of some options (e.g. -load_all) you end up telling the native linker to bring everything into the application (required or not);

Now debug builds are larger (because we use a soft debugger, i.e. additional code is injected) but release builds might still fit under the limit. Now being limited to debug on 64bits is not ideal, so I strongly suggest you check your settings to ensure your application size is minimized and, if needed, take additional steps (e.g. "Link All" and skipping a few assemblies you have no control over) to minimize the size.

[1] Have you double check that you have "Link SDK" enabled on your project ? I can't tell from your logs.

[jwholme2]

Thank you. The build in question is using a debug | iPhone build configuration. And yes, the Linker Behavior is set to Link Framework SDKs only for the debug | iPhone configuration.

[jwholme2]

FWIW I did try to do a debug device build with the Link All setting turned on, but it had a similar linking error as when I just had the SDK enabled.

I'm rerunning it a second time with the Additional mtouch argument verbosity flags you suggested.

[jwholme2]

Turning LinkAll on gave me a similar linking-related error:

https://gist.github.com/jwholme2/c68abca54377767adc24f45f8bb1dfb1

[spouliot]

error: can not encode offset '0x19D3D78' in resulting scattered relocation.

yeah, no doubt it's the native linking limitation on 32bits.

I'm surprised Link All did not help (likely it did but not enough). Your application must be quite large (by itself) or include a lot of native dependencies.

[jwholme2]

For context: Previous good builds were showing as ~122 MB in AppCenter. Switching the builds to arm64 reduced it to ~65MB.

[spouliot]

That's quite a lot of code for an executable. Not knowing the app I can't say if it's normal (or not). If you use a lot of 3rd party libraries then you should look closely at your native linker settings to ensure they are optimal (and let the tool to it's job to remove unused code).

[jwholme2]

We do use a few 3rd party libraries. Are you referring to the linker settings outlined here? https://docs.microsoft.com/en-us/xamarin/ios/deploy-test/linker?tabs=windows#controlling-the-linker

[spouliot]

No, that's the managed linker - so it only works on managed code.

If you have a lot of 3rd parties with native libraries (e.g. bindings) then you want to look at the arguments (that each of them can provide) to the native linker (ld usually called thru clang).

https://gist.github.com/jwholme2/8a12eb9ceb87c897da0e49a587187f09#file-gistfile1-txt-L975
lists all the pieces that you're asking the native linker to glue together.

In particular look at the -force_load used 4 times since

-all_load and -force_load act a link time, not at load time. -all_load and -force_load tell the linker to link the entire static archive in the final executable, even if the linker thinks that parts of the archive are unused. Cocoanetics/DTCoreText#169 (comment)

[jwholme2]

Thank you for your help!!! I will check our native linker settings.

[jwholme2]

If it's OK, I will mark as closed.

[jwholme2]

Hi again @spouliot ! I've been trying to find a way to override the -force_load tag used by certain 3rd party assemblies with the native linker, but I'm not finding a way to do this. Any pointers?

I also tried removing one of the assemblies using -force_load referenced in the logs - Calabash, but that did not seem to help the build.