iOS test fails with "App is not signed"

[jkotas]

[18:49:34] dbug: ApplicationVerificationFailed: Failed to verify code signature of /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.TXkPbA/extracted/iOS.Device.Aot.Test.app : 0xe800801c (No code signature found.)
[18:49:34] dbug: error MT1006: Could not install the application '/tmp/helix/working/B17B099C/w/A7FD08E3/e/iOS.Device.Aot.Test.app' on the device '00008030-001D059A26D3402E': AMDeviceSecureInstallApplicationBundle returned: 0xe800801c.
[18:49:34] dbug: 
[18:49:34] dbug: AMDeviceSecureInstallApplicationBundle returned: 0xe800801c
[18:49:34] dbug: Process mlaunch exited with 1
[18:49:34] fail: Failed to install the application
                 App is not signed

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=885579
Build error leg or test failing: iOS.Device.Aot.Test.WorkItemExecution
Pull request: #110344

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "App is not signed",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=885579
Error message validated: [App is not signed]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 12/4/2024 2:09:44 PM UTC

Report

Build	Definition	Test	Pull Request
890088	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110532
889679	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#109508
889650	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution
889632	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110514
889597	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110518
889581	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110501
889548	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110515
889527	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110481
887292	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110123
889444	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110465
889443	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution
889383	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution	#110501
889289	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution
889250	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110342
889212	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110506
889192	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution	#110342
889140	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110033
889138	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution
889100	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110501
889095	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110503
889020	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#110477
888986	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#104906
888907	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution
888893	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110342
888589	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110491
888474	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110449
888810	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution	#110485
888803	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110481
888660	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110494
888785	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110486
888749	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#109431
888709	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110033
888706	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110465
888690	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110495
888694	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#109378
888644	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110479
888642	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110493
888625	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#101356
888617	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110448
888621	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110492
888566	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#110471
888626	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#101356
888563	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110484
888561	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110476
888553	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110486
888536	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution
888581	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110490
888569	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110485
888494	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110477
888537	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110064
888424	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#104906
888408	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110483
888417	dotnet/runtime	tvOS.Device.Aot.Test.WorkItemExecution	#110484
888317	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110478
888292	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#108941
888028	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#110468
888289	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#108941
888279	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110476
888363	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110481
888320	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110479
888242	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110475
888192	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110119
888145	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110471
888153	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110472
888112	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110448
888132	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110432
888135	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110469
888235	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110474
888088	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#110117
888078	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110467
888035	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#108513
888001	dotnet/runtime	Invariant.Tests.WorkItemExecution
887998	dotnet/runtime	System.Runtime.Tests.WorkItemExecution
887984	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#108732
887987	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110457
887979	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110392
887945	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution
887913	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110437
887859	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110465
887536	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110419
887708	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110432
887687	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110459
887707	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution
887679	dotnet/runtime	iOS.Device.ExportManagedSymbols.Test.WorkItemExecution	#110458
887639	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110457
887602	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#109707
887616	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110455
887598	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110449
887545	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#109087
887530	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution
887518	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110452
887506	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110451
887478	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110322
887455	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#102553
887472	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#104853
887442	dotnet/runtime	iOS.Device.Aot.Test.WorkItemExecution	#110449
887390	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110378
887491	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#109320
886558	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#110416
887459	dotnet/runtime	iOS.Device.LibraryMode.Test.WorkItemExecution	#109707
Displaying 100 of 156 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	156	156

[ivanpovazan]

This seems like an infra issue.
Going through the log we first hit this:

Signing bundle /tmp/helix/working/B17B099C/w/A7FD08E3/e/iOS.Device.Aot.Test.app
    "/Users/helix-runner/Library/Keychains/signing-certs.keychain-db"
security: SecKeychainUnlock signing-certs.keychain-db: The user name or passphrase you entered is not correct.

Which fails to sign the app, and later the installation gets refused.

@AlitzelMendez we had this reported before: #101830 which got resolved with: dotnet/dnceng#2667 as it affected only few machines.
The error messages look the same, so I assume we are experiencing the same problem here now on different machines, could you please investigate?
(For reference the log I linked is collected on: DNCENGMAC127)

[AlitzelMendez]

@ivanpovazan I am no longer on the team but @dotnet/dnceng should be able to help you with this 😄

[markples]

This seems to be more widespread than #101830. For example, https://dev.azure.com/dnceng-public/public/_build/results?buildId=888292&view=results hit it on DNCENGMAC017, DNCENGMAC046, DNCENGMAC048, DNCENGMAC103, DNCENGMAC111, DNCENGMAC117, DNCENGMAC120, DNCENGMAC124, and DNCENGMAC128.

[premun]

Our team was rotating the signing certificates last week and this is a possible fallout. I will have a look at what's wrong.

[premun]

I have identified the problem - the script that was used for refreshing the Apple certificates failed to store the keychain password.
I am now rolling out the new certificates to all iPhone and AppleTV queues again.