iOS objc FlutterChannelsTest tests flaky on an arm64 build

[dnfield]

I'm noticing failures of testCallsNotifyLowMemory in flutter/engine#41685. When I try to run it locally, I get different failures or sometimes no failures at all.

Not clear yet whether this is just an issue with my patch.

[jmagman]

I suspect a crash caused by your PR:

IosUnitTests(38439) MallocStackLogging: process 38430 no longer exists, stack logs deleted from /tmp/stack-logs.38430.1045f8000.IosUnitTests.uLQtmy.index

Restarting after unexpected exit, crash, or test timeout in -[FlutterEnginePlatformViewTest testCallsNotifyLowMemory]; summary will include totals from previous launches.

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8782093625669830065/+/u/test:_Tests_for_ios_debug_sim_arm64/stdout

Unfortunately we don't upload the crash logs from those tests in LUCI yet, that's being worked on though #125823

[cyanglaz]

Find local xcresult file for your local test failure log

https://stackoverflow.com/questions/69096308/where-can-i-find-the-xcresult-logs-from-my-application-in-an-xcode-ui-test

[dnfield]

The local xcresult file just says the test failed, I get no extra logs :\ And the test seems to pass sometimes... This could very well be from my change though, I'll keep looking into it.

[dnfield]

Here's a local failure that I'm not seeing in CI on my PR, but does happen flakily:

Something in the FlutterChannelsTest suite is causing this (@gaaclarke in case you've seen this before):

Thread 7 Crashed::  Dispatch queue: com.apple.dt.xctest.default-debug-log-handler
0   libsystem_kernel.dylib        	       0x1b17e8310 __abort_with_payload + 8
1   libsystem_kernel.dylib        	       0x1b180bd94 abort_with_payload_wrapper_internal + 100
2   libsystem_kernel.dylib        	       0x1b180bd30 abort_with_reason + 28
3   libobjc.A.dylib               	       0x1800510cc _objc_fatalv(unsigned long long, unsigned long long, char const*, char*) + 112
4   libobjc.A.dylib               	       0x18005105c _objc_fatal(char const*, ...) + 28
5   libobjc.A.dylib               	       0x180035a18 getMaybeUnrealizedNonMetaClass(objc_class*, objc_object*) + 700
6   libobjc.A.dylib               	       0x18004317c resolveMethod_locked(objc_object*, objc_selector*, objc_class*, int) + 124
7   libobjc.A.dylib               	       0x1800318b4 class_getMethodImplementation + 36
8   Foundation                    	       0x180ab70e4 _classOverridesSelector + 56
9   Foundation                    	       0x180ab7070 -[NSCoder validateClassSupportsSecureCoding:] + 336
10  Foundation                    	       0x180b38a8c _encodeObject + 344
11  Foundation                    	       0x180b37974 +[NSKeyedArchiver archivedDataWithRootObject:requiringSecureCoding:error:] + 88
12  DTXConnectionServices         	       0x1049ebb0c 0x1049d8000 + 80652
13  DTXConnectionServices         	       0x1049ebf30 0x1049d8000 + 81712
14  DTXConnectionServices         	       0x1049da788 0x1049d8000 + 10120
15  CoreFoundation                	       0x18043afec ___forwarding___ + 740
16  CoreFoundation                	       0x18043d50c _CF_forwarding_prep_0 + 92
17  XCTestCore                    	       0x1056e85f8 -[XCTRunnerIDESession logDebugMessage:] + 56
18  XCTestCore                    	       0x1056ceb18 -[XCTDefaultDebugLogHandler _queue_flushDebugMessageBufferWithBlock:] + 204
19  XCTestCore                    	       0x1056ceeb4 __45-[XCTDefaultDebugLogHandler logDebugMessage:]_block_invoke + 248
20  libdispatch.dylib             	       0x180132ee4 _dispatch_call_block_and_release + 24
21  libdispatch.dylib             	       0x180134708 _dispatch_client_callout + 16
22  libdispatch.dylib             	       0x18013c77c _dispatch_lane_serial_drain + 776
23  libdispatch.dylib             	       0x18013d414 _dispatch_lane_invoke + 448
24  libdispatch.dylib             	       0x180149608 _dispatch_workloop_worker_thread + 768
25  libsystem_pthread.dylib       	       0x1b1834878 _pthread_wqthread + 284
26  libsystem_pthread.dylib       	       0x1b183363c start_wqthread + 8

[dnfield]

Ok, the reason I wasn't seeing the same test failure as remote was because I had a stale build of ios_tests_flutter. However, I do see flakes locally on an arm64 machine of the FlutterChannelsTest suite.

[dnfield]

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8782009540418918689/+/u/test:_Tests_for_ios_debug_sim_arm64/stdout

https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20mac_unopt/946/overview are some more recent examples

[jmagman]

Example of flaky test not from that engine PR from duplicate #126084:
https://ci.chromium.org/ui/p/flutter/builders/try/Linux%20mac_unopt/946/overview

Test Case '-[FlutterChannelsTest testBasicMessageChannelInvokeHandlerAfterChannelReleased]' started.
objc[64626]: no class for metaclass 0x102e7c0f8
IosUnitTests(64641,0x10094c580) malloc: enabling scribbling to detect mods to free blocks
IosUnitTests(64641) MallocStackLogging: could not tag MSL-related memory as no_footprint, so those pages will be included in process footprint - (null)
IosUnitTests(64641) MallocStackLogging: stack logs being written to /tmp/stack-logs.64641.1008bc000.IosUnitTests.rFtmpe.index
IosUnitTests(64641) MallocStackLogging: recording malloc and VM allocation stacks to disk using standard recorder
IosUnitTests(64641) MallocStackLogging: process 64626 no longer exists, stack logs deleted from /tmp/stack-logs.64626.102d58000.IosUnitTests.ntNGdZ.index

Restarting after unexpected exit, crash, or test timeout in -[FlutterChannelsTest testBasicMessageChannelInvokeHandlerAfterChannelReleased]; summary will include totals from previous launches.

Failing tests:
	-[FlutterChannelsTest testBasicMessageChannelInvokeHandlerAfterChannelReleased]
	-[FlutterChannelsTest testBasicMessageChannelTaskQueue]

@cyanglaz flutter/engine#41647 would be really helpful to land to diagnose this.