off topic: linking two Mac Studio together to fit larger models

[gileneusz]

Hi, I just came across an idea of linking two Mac Studio 192GB with Thunderbolt bridge to fit larger models. Has anyone tried this before? Would it be even possible and would Thunderbolt bridge speed be a bottleneck in this case?

[netrunnereve]

Have you tried setting up MPI? Since the Thunderbolt Bridge is treated as an IP connection it may work.

[gileneusz]

I haven't tried this connection myself; I'm merely suggesting an idea that could be tested with any pair of Macs equipped with a Thunderbolt port. This would allow for an assessment of the throughput and latency (T/S) output, which could then be used to predict the performance when connecting machines with 192+192 GB of RAM. However, I'm not deeply versed in programming to implement this myself; I'm just putting the idea forward.

[izard]

I checked that a while ago with 2x 128GB Mac Studios over direct 10GB link. Everything worked with llama.cpp/MPI, and performance was reasonable, both systems were memory throughput limited at 1-batch, so 10GB NIC throughput was enough.

[nachoal]

This idea is starting to gain traction in the twitter world, see this tweet, @izard could you share a tutorial on how to set it up?

[netrunnereve]

If you're using regular Ethernet it should work with the standard MPI setup. Did you try the tutorial in the README?

[ggerganov]

There is also the RPC backend by @rgerganov: #6829

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.