FirebaseCrashlytics run vs upload-symbols -- missing optional dsyms

[dloic]

[REQUIRED] Step 1: Describe your environment

• Xcode version: 11.2.1
• Firebase SDK version: 6.17.0, 6.24.0
• Firebase Component: Crashlytics
• Component version: 6.17.0, 6.24.0
• Installation method: Carthage

[REQUIRED] Step 2: Describe the problem

We're seeing missing optional dsym for our apps since a month now and we can't explain why it's happening. As said in the Firebase console, it's only affecting result and grouping, but still we would like to understand why it's like this and what did broke our integration. We've seen this behavior on ad-hoc build without bitcode (we generate dsym) and also on store build (bitcode enabled, dsym pulled from ASC).

Steps to reproduce:

To reproduce the issue, I've run a test outside of our CI, to ensure the problem is not there.
Initial steps to ensure app is compiled and dsym are uploaded:

• Xcode clean
• Xcode build for device arch
• Xcode archive

As seen in screenshot below, it seems everything is fine during symbols upload:

In the app, we have a way from our debug menu to trigger a crash, to easily test our Crashlytics integration is working from release to release. After the build is installed on a device (ad-hoc distribution), I simply open the debug menu and crash the app. Result in Firebase console:

This is telling us we're missing the AA71654B-XXX dsym, with turns to be the one generated for Eureka third-party. Having a look within the Xcode archive and using dwarfdump, I can confirm this dsym has been properly generated:

I then manually upload-symbols. End result is now the console is able to properly symbolicate new crashes (not older ones):

Does it means that during build, upload was somehow not successful? Or we're missing something in our integration?
As for our full update path, we were using Fabric Crashlytics for a while, and we did migrate to FirebaseCrashlytics with Firebase 6.17.0 in Feb. I'm currently updating Firebase to latest (6.24.0), but it seems the issue is still there.

Thanks for your help!

[dloic]

Hi @TKBurner , any news/feedback regarding my issue?

[eric-chamberlain-ck]

We've encountered the same issue. It appears that upload-symbols with the —build-phase flag only uploads the dSYM for the build target, not all the dSYMs in DWARF_DSYM_FOLDER_PATH.

The workaround is to call it manually with something like ${PROJECT_DIR}/../scripts/FirebaseCrashlytics/upload-symbols -d -p ios -gsp ${BUILT_PRODUCTS_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}/GoogleService-Info.plist ${DWARF_DSYM_FOLDER_PATH}

[dloic]

@eric-chamberlain-ck Thanks for the info. As said in my initial comment, it's also working for us if upload manually.

Seems there is an issue in upload-symbols binary, because as you said, looking at https://github.com/firebase/firebase-ios-sdk/blob/master/Crashlytics/run the only diff is --build-phase option which seems to be some kind of shortcut to avoid giving platform and path to service info plist. But maybe it's doing more than this?

Can someone from the Firebase team confirms this?

[TKBurner]

@samedson is going to take a look at this.

[samedson]

@dloic you're right: the --build-phase flag uses environment variables that Xcode defines during the build to determine the app's platform and path to the plist. But in addition, it is also able to get all the dSYM locations from those environment variables.

When using upload-symbols directly, it should take either a directory or a list of dSYM paths, so you'll want to pass all your dSYMs paths to upload-symbols separated by spaces.

[nickprkins]

@samedson so why is the build phase script not uploading these optional DSYMs? I have a ton of optional DSYMs that need to be uploaded. I tried downloading them from AppStoreConnect and from our CS and uploading them and it finds none of these optional DSYMs that Firebase says are missing. Something is not working properly with the way FirebaseCrashlytics wants these setup to upload. Losing confidence in this product.

[samedson]

Ah so let me clarify a few things about dSYMs in general.

• Usually your app has a main dSYM that's put in the Products directory. Crashlytics is usually able to gather this dSYM because it's clearly defined in Xcode's Build Phase environment variables, that upload-symbols reads from. This is the "Required" dSYM in Crashlytics.

  • When you download a dSYM from iTunesConnect, it will be this "Required" dSYM.

• When there are "Optional" dSYMs in Crashlytics, this usually indicates there are separate SDKs included in your app. There are a few possibilities on where to find these:

  • In some cases the dSYM gets generated for the SDK into the Products directory as well, in which case upload-symbols should find it.
  • Sometimes if the .framework folder for the SDK has the dSYM inside of it, it will need to be uploaded manually via upload-symbols separately. upload-symbols can have trouble finding these dSYMs because they are not usually referenced by Xcode's build environment variables.
  • Sometimes you have an SDK with stripped symbols and no .dSYM. In this situation there's now way to get symbolicated crashes for those lines of code because the SDK developer purposefully hid all symbol information.

The easiest way to find a missing dSYM file is to run:

mdfind "com_apple_xcode_dsym_uuids == UUID"

In general, Xcode determines if dSYMs are generated, and where they're put on the filesystem. Crashlytics has no control over this and won't modify Xcode projects.

[dloic]

Thanks for the info @samedson. As mentioned by others, we're definitely seeing a new behavior. Do you confirm?

Could it be related to the new Crashlytics SDK?

As mentioned by @nickprkins we're seeing the same issue for AppStore build, none of the dsym downloaded manually from ASC are matching the missing optional dsyms.

Could this be also related to Carthage? @eric-chamberlain-ck @nickprkins are you using Carthage?

[eric-chamberlain-ck]

We are using Carthage and Fastlane. All our dSYMs are in the same folder at the end of the build, using the —build-phase flag only uploads the app dSYM and not all the dSYMs in the folder.

We ended up moving the upload out of our build phases, manually specifying the flags for upload-symbols, and trigger the upload later in our CI flow.

[dloic]

@samedson I got no feedback from you. Should we do a hack and call upload-symbols, as described above? And thus no longer follow official documentation?

[samedson]

@dloic yeah for now I would manually call upload-symbols by passing all the dSYMs to it. I haven't yet had time to figure out what is going wrong.