Different behavior on iphone 12 and Macbook Pro of the same model on the same images

[MikeMpapa]

❓Question

I have trained a custom object detector to detect a specific object of interest through turicreate and then converted my final model to coreML.

I observed that the model behaves differently when integrated in the iphone app and when tested on my laptop. In particular, both the turicreate model and the extracted coreML version of it seem to be more reliable when tested on my MacBook against the same coreML model as part of the iOS app when tested on exactly the same images.

The on-device copy of the coreML model gives more false positives than when tested locally on the same images.

Any thoughts or advise??

Thanks a lot in advance!!

[TobyRoseman]

How large are the differences between the Core ML model on macOS vs iOS? Can you find a single image where the predictions differ significantly?

[MikeMpapa]

Well...there are several images where I get multiple detections when tested on iphone and just a single detection when tested on the macbook. Did I answer your question? Do you mean something else by 'large differences'?

[srikris]

Usually this has to do with FP32/FP16. Can you try running it on the device with this flag: https://developer.apple.com/documentation/coreml/mlpredictionoptions/2921288-usescpuonly. This will help eliminate if the issue is during conversion of precision related.

[MikeMpapa]

Here is a small example of the outputs we get on different model evaluations using the same model. Overall :

• Results look very similar between iOS accelerated, CPU only and MAC-iOS Simulator.

• Some differences are observed between coreMLtools and the above

  21.02.26-07.58.31-.neg.png shows big difference between turicreate and coreml model (on mac and iOS), confidence value
  much lower. From .53 on turicreate to .50 in coremltools to .4 ish on simulator and ios

• Larger differences are observed between turicreate and the coreML version of the model

  21.01.19-15.16.06-R.png shows big difference, turicreate .77 to .43 on coremltools, coreml on mac and ios are .42
  21.01.20-16.25.10-.multi.png only one detection with turicreate but two in all other cases

---

iOS iphone 12 mini accelerated (confidence is last)

21.01.19-15.16.06-R.png
0.21484375, 0.16455078125, 0.65380859375,0.84033203125, 0.41723633

21.02.26-07.58.31-.neg.png
0.2164306640625, 0.36602783203125, 0.3538818359375,0.50506591796875, 0.43310547

21.02.25-08.37.21-.multi.png
0.4359130859375, 0.24725341796875, 0.5689697265625,0.42169189453125, 0.9057617

21.01.20-16.25.10-.multi.png
0.50140380859375, 0.34698486328125, 0.63336181640625,0.50848388671875, 0.82470703
0.53131103515625, 0.374664306640625, 0.60931396484375,0.484710693359375, 0.28222656

21.02.26-07.56.18-L.C266.768.247.247.png
0.2906494140625, 0.33770751953125, 0.4207763671875,0.48455810546875, 0.91503906

---

iOS iphone 12mini cpu ONLY (confidence is last):

21.01.19-15.16.06-R.png
0.21502652764320374, 0.16459670662879944, 0.6538605391979218,0.8405994474887848, 0.4173663

21.02.26-07.58.31-.neg.png
0.2166622132062912, 0.365869864821434, 0.3540227562189102,0.5049778372049332, 0.4325431

21.02.25-08.37.21-.multi.png
0.43630602210760117, 0.24669453501701355, 0.569352962076664,0.4211321175098419, 0.90601486

21.01.20-16.25.10-.multi.png
0.5015846788883209, 0.3466792479157448, 0.6335439383983612,0.5082250460982323, 0.8256004
0.5314093753695488, 0.37469274550676346, 0.6094856336712837,0.4847322031855583, 0.28201136

21.02.26-07.56.18-L.C266.768.247.247.png
0.2907545417547226, 0.3375038132071495, 0.4208303242921829,0.4843718782067299, 0.9155101

---

simulator - Mac (confidence is last)

21.01.19-15.16.06-R.png
0.21406057476997375, 0.16404366493225098, 0.654240757226944,0.8409333229064941, 0.42460945

21.02.26-07.58.31-.neg.png
0.21639683097600937, 0.3661840632557869, 0.3540252223610878,0.504875473678112, 0.434828

21.02.25-08.37.21-.multi.png
0.43610650300979614, 0.24673673510551453, 0.5693313479423523,0.42081204056739807, 0.90558153

21.01.20-16.25.10-.multi.png
0.5014769583940506, 0.34683461487293243, 0.6328367441892624,0.5077475756406784, 0.8279768
0.53095643222332, 0.37442685663700104, 0.6092924624681473,0.48459766805171967, 0.28763768

21.02.26-07.56.18-L.C266.768.247.247.png
0.2905817776918411, 0.3375796154141426, 0.4208042472600937,0.4847891256213188, 0.913991

---

coreMLtools in python on Mac (after conversion from turicreate)

21.01.19-15.16.06-R.png
{'coordinates': array([[0.43505859, 0.49707031, 0.4453125 , 0.67578125]]), 'confidence': array([[0.43725586]])}

21.02.26-07.58.31-.neg.png
{'coordinates': array([[0.28637695, 0.56396484, 0.13842773, 0.13696289]]), 'confidence': array([[0.50634766]])}

21.02.25-08.37.21-.multi.png
{'coordinates': array([[0.50244141, 0.66552734, 0.13256836, 0.1739502 ]]), 'confidence': array([[0.90136719]])}

21.01.20-16.25.10-.multi.png
{'coordinates': array([[0.56738281, 0.57324219, 0.1328125 , 0.16015625],
[0.56982422, 0.57128906, 0.08013916, 0.10992432]]), 'confidence': array([[0.83154297],
[0.28198242]])}

21.02.26-07.56.18-L.C266.768.247.247.png
{'coordinates': array([[0.35546875, 0.58837891, 0.12902832, 0.14746094]]), 'confidence': array([[0.91162109]])}

---

TuriCreate in python on Mac

21.01.19-15.16.06-R.png
[[{'label': 'train', 'type': 'rectangle', 'coordinates': {'x': 117.98262441158295, 'y': 137.04818487167358, 'width': 125.38967895507812, 'height': 195.7066192626953}, 'confidence': 0.7685595154762268}]]

21.02.26-07.58.31-.neg.png
[[{'label': 'train', 'type': 'rectangle', 'coordinates': {'x': 313.00806283950806, 'y': 811.2643933296204, 'width': 169.337890625, 'height': 196.4040069580078}, 'confidence': 0.5307837724685669}]]

21.02.25-08.37.21-.multi.png
[[{'label': 'train', 'type': 'rectangle', 'coordinates': {'x': 539.6951374411583, 'y': 966.9835352897644, 'width': 141.5130157470703, 'height': 241.9056854248047}, 'confidence': 0.8975157737731934}]]

21.01.20-16.25.10-.multi.png
[[{'label': 'train', 'type': 'rectangle', 'coordinates': {'x': 607.6200476288795, 'y': 824.8291289806366, 'width': 141.012939453125, 'height': 253.7781524658203}, 'confidence': 0.8388118147850037}]]

21.02.26-07.56.18-L.C266.768.247.247.png
[[{'label': 'train', 'type': 'rectangle', 'coordinates': {'x': 387.03433603048325, 'y': 847.4364173412323, 'width': 147.85247802734375, 'height': 212.3245086669922}, 'confidence': 0.8649961352348328}]]

[TobyRoseman]

Is this still an issue with the latest version of macOS and iOS?