[Open-to-community] Benchmark swift-coreml-diffusers on different Mac hardware

[Vaibhavs10]

Hey hey,

We are on a mission to provide a first-class, one-click solution to blazingly fast diffusers inference on Mac. In order for us to get a better idea of our framework, we'd like to get inference time benchmarks for the app.

Currently, we are explicitly looking for benchmarks on:

• M1 Pro - @tcapelle, @emwdx, @Pindar777
• M1 Pro (6/14/16) - @abazlinton
• M2 Pro - @tcapelle, @mja, @SerCeMan
• M2 Max - @Tz-H, @lovelace

You can do so by following the below steps:

1. Download the latest version of the Diffusers app from the App store.
2. Select one of the three options in the Advanced
3. Insert a random prompt for e.g. A Labrador playing in the fields.
4. Run inference and make a note of the time taken for inference.

Note: Do make sure to run inference multiple times as the framework sometimes requires to prepare the weights in order to run it in the most efficient way possible.

Ping @pcuenca and @Vaibhavs10 for any queries or questions!

Happy diffusing 🧨

[tcapelle]

I can do the M2Pro Mac Mini!

[pcuenca]

I can do the M2Pro Mac Mini!

Cool, assigned it to you above :)

[tcapelle]

With default settings with 25 steps:

Macbook Pro 14" with M1 Pro GPU 16 Cores - 16GB of ram - 8 perf cores

• ANE: 15.4s, 15.2, 15.2
• GPU: 13.7s, 13.9s, 13.7s (Using less than 4GB of ram 🤯)
• GPU + ANE: 15.4, 15.2, 15.4

Mac Mini with M2 Pro GPU 16 Cores - 16GB of ram - 6 perf cores

• ANE: For some reason, on this machine the ANE was the default: 10.4, 10.3, 10.4 (no ram usage reported?!)
• GPU: 12.4s, 12.3s, 12.3s
• ANE+GPU: 10.9, 10.8, 10.8

[pcuenca]

Thanks a lot @tcapelle that's super helpful!

For some reason, on this machine the ANE was the default

Yeah, we have a simple rule (based on the number of performance cores, which is a good proxy for the rest of the hardware). It looks like it worked in both your computers, didn't it? (the best option was selected by default).

A couple of questions, if you can.

• How many performance cores does each computer have? (I find sysctl hw.perflevel0.physicalcpu to be easy).
• What model did you test? Relative performance is usually consistent across models.

The ANE+GPU performance is very close in both computers! I'm expecting ANE+GPU to beat just ANE in some of the MBP M2 Pro combinations.

[tcapelle]

I used default settings, so it's sd-base-2.0

[tcapelle]

do you know a trick to query how many GPU cores the machine has?
Would be really cool to retrieve this info programatically so we can log it to the wandb info:

[pcuenca]

I suppose terminal is ok:

ioreg -l | grep gpu-core-count | tail -1 | awk -F"=\ " '{print $NF}'

(Only produces results on Apple Silicon)

[tcapelle]

there is also this thingy: https://github.com/tlkh/asitop

[pcuenca]

there is also this thingy: https://github.com/tlkh/asitop

Oh interesting. This is what they do: https://github.com/tlkh/asitop/blob/main/asitop/utils.py#L123

[emwdx]

Same config as tcapelle above for comparison.

Default settings with 25 steps, Macbook Pro 14" with M1 Pro GPU 16 Cores - 16GB of ram - 8 perf cores

ANE: 15.2, 15.1, 15.3
GPU: 13.9, 13.7, 13.7
GPU + ANE: 14.2, 14.5, 14.4

Similar results as above, so that's cool.

[pcuenca]

Thanks a lot @emwdx! I think the app should have selected the best option (GPU) for you, is that correct?

Interestingly, your GPU+ANE combination is better than @tcapelle's (although still not better than just GPU).

[emwdx]

It did automatically select the GPU, yes :)

…

On Sat, Feb 25, 2023 at 1:16 PM Pedro Cuenca ***@***.***> wrote: Thanks a lot @emwdx <https://github.com/emwdx>! I think the app should have selected the best option (GPU) for you, is that correct? — Reply to this email directly, view it on GitHub <#31 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALV7N3DWBME4YK5X3LJE4TWZIAZVANCNFSM6AAAAAAVGWC7KA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[abazlinton]

Amazing work huggingface team ❤️!

Here are mine -

14" MacBook M1 Pro - 14 GPU cores / 6 performance cores - All settings default (SD 2-base)

---

ANE: 15.2, 15.2, 15.2
GPU: 15.1, 15.1, 15.2
ANE+GPU: 14.4, 14.5, 14.4

[Tz-H]

14" MacBook M2 Max - 64 GB - 30 cores

hw.perflevel0.physicalcpu: 8

Settings

• Models: stabilityai/stable-diffusion-2-1-base
• Prompts: A Labrador playing in the fields
• Steps: 25
• Seed: -1

Result

• GPU: 7.7, 7.7, 7.6
• ANE: 10.3, 10.3, 10.3
• GPU + ANE: 10.6, 10.6, 10.7