GLTFPackedDataForAccessor memcpy crashes

[lenkawell]

We've been seeing these GLTFKit2 crashes randomly on user devices since August of this year. Unfortunately the Apple crash logs don't include anything useful like which GLTF model was being loaded. The crash always seems to be the memcpy. I have 5 .crash files if you would like them. Crash last evening happened to me while I was loading a test scene with 72 models on an iPad Pro M4 with 8GB of memory. When I re-ran the test, memory check shows 2.6 GB in use by our app.

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Subtype: KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Codes: 0x0000000000000001, 0x0000000000000000
VM Region Info: 0 is not in any region.  Bytes before following region: 4373774336
      REGION TYPE                 START - END      [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                   104b28000-104de4000 [ 2800K] r-x/r-x SM=COW  /var/containers/Bundle/Application/08275B3E-E57D-4256-BB9C-5744CF8BC975/Movie Maker 3D.app/Movie Maker 3D
Termination Reason: SIGNAL 11 Segmentation fault: 11
Terminating Process: exc handler [3356]

Triggered by Thread:  0



Thread 0 Crashed:
0   libsystem_platform.dylib      	0x00000002253c324c 0x2253c0000 + 12876
1   GLTFKit2                      	0x0000000105314c60 GLTFPackedDataForAccessor + 304 (GLTFAsset.m:104)
2   GLTFKit2                      	0x00000001052fe910 GLTFSCNGeometrySourceForAccessor + 124 (GLTFSceneKit.m:352)
3   GLTFKit2                      	0x00000001052fae14 -[GLTFSCNSceneSource convertAsset] + 8392 (GLTFSceneKit.m:835)
4   GLTFKit2                      	0x00000001052f8aa8 -[GLTFSCNSceneSource initWithAsset:] + 96 (GLTFSceneKit.m:529)
5   Movie Maker 3D                	0x0000000104bedc88 @nonobjc GLTFSCNSceneSource.init(asset:) + 16 (/<compiler-generated>:0)
6   Movie Maker 3D                	0x0000000104bedc88 GLTFSCNSceneSource.__allocating_init(asset:) + 28 (SceneObject+LoadModel.swift:399)
7   Movie Maker 3D                	0x0000000104bedc88 specialized closure #1 in closure #1 in closure #1 in static SceneObject.loadModelAsset(url:isUserAsset:completion:) + 76
8   Movie Maker 3D                	0x0000000104cc6574 thunk for @escaping @callee_guaranteed () -> () + 28 (/<compiler-generated>:0)
9   Movie Maker 3D                	0x0000000104b339c4 +[SwiftExceptionHandler catchException:error:] + 40 (SwiftExceptionHandler.m:18)
10  Movie Maker 3D                	0x0000000104bed964 closure #1 in closure #1 in static SceneObject.loadModelAsset(url:isUserAsset:completion:) + 1104 (SceneObject+LoadModel.swift:396)
11  Movie Maker 3D                	0x0000000104cc6574 thunk for @escaping @callee_guaranteed () -> () + 28 (/<compiler-generated>:0)
12  libdispatch.dylib             	0x00000001a26ce248 _dispatch_call_block_and_release + 32 (init.c:1549)
13  libdispatch.dylib             	0x00000001a26cffa8 _dispatch_client_callout + 20 (object.m:576)
14  libdispatch.dylib             	0x00000001a26dea34 _dispatch_main_queue_drain + 984 (queue.c:8093)
15  libdispatch.dylib             	0x00000001a26de64c _dispatch_main_queue_callback_4CF + 44 (queue.c:8253)
16  CoreFoundation                	0x000000019a999bbc __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16 (CFRunLoop.c:1793)
17  CoreFoundation                	0x000000019a9961b0 __CFRunLoopRun + 1996 (CFRunLoop.c:3163)
18  CoreFoundation                	0x000000019a9e8274 CFRunLoopRunSpecific + 588 (CFRunLoop.c:3434)
19  GraphicsServices              	0x00000001e7b314c0 GSEventRunModal + 164 (GSEvent.c:2196)
20  UIKitCore                     	0x000000019d52a77c -[UIApplication _run] + 816 (UIApplication.m:3846)
21  UIKitCore                     	0x000000019d150e64 UIApplicationMain + 340 (UIApplication.m:5503)
22  Movie Maker 3D                	0x0000000104de7c8c main + 64 (AppDelegate.swift:56)
23  dyld                          	0x00000001c0ba4de8 start + 2724 (dyldMain.cpp:1338)

[warrenm]

That's a null pointer dereference, so it could indicate that a previous allocation failed, though that's not the only possibility.

2.6GiB is a pretty high water mark for some devices. I'll try to run some stress tests to see if I can reproduce this.

[warrenm]

One thing I notice is that if an app consumes a large amount of memory and the system has already identified that it is in a memory pressured state, it will most often terminate the app and generate a jetsam report. Based on the fact that you're getting an ordinary crash report instead, I suspect it's not just memory pressure at play.

If I request an absurd amount of memory at once from malloc, say 16GiB, it returns null without causing immediate termination. This quickly leads to a crash down in _platform_memmove when the null pointer is used as the destination of the copy.

This leads me to suspect one of two things: either you're dealing with a corrupt asset that causes an erroneously huge allocation to be requested, or perhaps there's some kind of heap corruption that's affecting the allocation size, or both.

If you ever repro this with a debugger attached, the most interesting bits of data would be: the requested size of the allocation, and whether it in fact results in a null pointer being returned from malloc. I can't currently reproduce this with any collection of assets I have on-hand.

[lenkawell]

I agree with you that typically iOS will abort the app when it runs out of memory with no crash logging done. I have 14 GLTFPackedDataForAccessor crash logs for the latest app version. About 5 of them include a "VM - (arg = 0x3) mach_vm_allocate_kernel failed within call to vm_map_enter" in the log, which may or may not indicate a previous memory allocation failure. Though the rest of the logs do not include any Kernel Triage info. I haven't had it crash with the debugger again so I cannot give you a model or scene that crashes at this point.

How about adding a null check before the memmove that I can catch and at least not crash the app and hopefully log the model that is causing the crash?

[lenkawell]

Your memory allocation failure checks and logging look like they'll be helpful. I've pulled them and I'm testing them now. Assuming they pass the tests, we'll include them in our latest app update. Thanks very much.

[warrenm]

Sure thing. They don’t cover all failure cases, but they may turn up some interesting data, and should prevent a significant set of possible crashes.