metal : key the fa-vec tuned table by family instead of SKU

[forforever73]

Overview

taking the per-family direction that came out of discussion #27668.

fa_vec_tuned_table is rekeyed from device_id to gpu_family: 3884 rows across 19 SKUs become 1033 rows across 4 Apple GPU families. fa_vec_pick loses its device_id argument and its representative-SKU retry, so the lookup is one level instead of two. The tool still emits per-SKU rows.

End-to-end on M4 Max

target + draft	KV	context	master tok/s	PR tok/s	change
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	f16	2048	60.25	60.28	1.001×
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	f16	4096	58.08	58.12	1.001×
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	f16	16384	50.90	50.94	1.001×
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	f16	32768	43.16	43.19	1.001×
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	q4_0	2048	58.65	58.69	1.001×
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	q4_0	4096	56.22	56.24	1.000×
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	q4_0	16384	45.59	45.90	1.007×
Llama-3.1-8B-Q4_K_M + Llama-3.2-1B-f16	q4_0	32768	36.37	36.96	1.016×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	f16	2048	93.88	95.74	1.020×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	f16	4096	86.61	87.47	1.010×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	f16	16384	59.16	59.23	1.001×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	f16	32768	41.29	41.38	1.002×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	q4_0	2048	88.96	90.25	1.015×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	q4_0	4096	79.50	79.48	1.000×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	q4_0	16384	49.27	49.69	1.008×
Qwen3-30B-A3B-Q4_K_M + Qwen3-0.6B-f16	q4_0	32768	32.62	33.20	1.018×
Qwen3.6-27B-MTP-Q8_0 (mtp)	f16	2048	28.29	28.44	1.005×
Qwen3.6-27B-MTP-Q8_0 (mtp)	f16	4096	27.82	27.98	1.006×
Qwen3.6-27B-MTP-Q8_0 (mtp)	f16	16384	28.04	28.19	1.005×
Qwen3.6-27B-MTP-Q8_0 (mtp)	f16	32768	27.19	27.32	1.005×
Qwen3.6-27B-MTP-Q8_0 (mtp)	q4_0	2048	28.11	28.09	0.999×
Qwen3.6-27B-MTP-Q8_0 (mtp)	q4_0	4096	26.35	26.34	1.000×
Qwen3.6-27B-MTP-Q8_0 (mtp)	q4_0	16384	27.88	27.87	1.000×
Qwen3.6-27B-MTP-Q8_0 (mtp)	q4_0	32768	24.55	24.54	1.000×
Qwen3.6-35BA3B-MTP (mtp)	f16	2048	111.02	111.14	1.001×
Qwen3.6-35BA3B-MTP (mtp)	f16	4096	110.78	110.97	1.002×
Qwen3.6-35BA3B-MTP (mtp)	f16	16384	107.26	107.52	1.002×
Qwen3.6-35BA3B-MTP (mtp)	f16	32768	103.05	103.16	1.001×
Qwen3.6-35BA3B-MTP (mtp)	q4_0	2048	102.94	103.04	1.001×
Qwen3.6-35BA3B-MTP (mtp)	q4_0	4096	113.65	113.97	1.003×
Qwen3.6-35BA3B-MTP (mtp)	q4_0	16384	99.05	99.23	1.002×
Qwen3.6-35BA3B-MTP (mtp)	q4_0	32768	78.76	78.88	1.002×

Nothing regresses: the lowest cell is 0.999×. The small gain is consistent and reproducible, but it does not come from the kernel — it comes from cheaper lookup.

find_cfg performs a linear table scan. For this workload, M4 Max often misses the exact bucket and falls back to the default config; the old path can scan the full table up to four times, including redundant retries through the family representative SKU (which is M4 Max itself for family 9).

Keying by family removes those redundant scans and searches a table 3.8× smaller, reducing lookup cost from 2149 ns to 282 ns per call. This also explains the small gains in cells where the selected config is unchanged.

None of this says the pooled rows pick faster kernels. In-sample they are 98.5% of per-device tuning at the attention level — marginally slower — which end to end is below what this test can resolve.

Requirements

• I have read and agree with the contributing guidelines.
• AI usage disclosure: Yes, claude was used for code and test.

[nikwen]

Thank you so much for working on this!

The fusion tests failed in CI (not sure if related):

0.03.409.038 I summary: 151 ok, 1 failed
0.03.409.038 W main: if the fusion counts are expected to change, run with --record to update the baseline:

./bin/test-llama-archs -o build-ci-models

A few comments inline. The rest looks good to me. 👍

[nikwen]

It would be great to have:

1. documentation on how to arrive at these unified values if we have the sweep results for each device
2. maybe a script for it (I doubt you arrived at them by hand :) )
3. a record of all historical sweep results and sweep logs somewhere (new file/directory in the repo?)

That would:

1. ensure that the method is documented somewhere and that we can apply it in the future when we need to tune new devices (M6 family 👀)
2. allow us to slightly improve the existing values by incorporating more devices into the mix (e.g., M5 and M5 Pro are missing from family 10 as far as I can tell)
3. allow us to update the tuning results if it becomes necessary in the future
4. allow people to reproduce the results for themselves

[forforever73]

You're right that we should have a reproducible workflow for this, especially for tuning new devices in the future. I'm planning to add the functionality for converting offline-collected sweep logs into table rows, as well as evaluating the performance of a newly generated table, to ggml-metal-tuning.

I'd prefer to keep this in ggml-metal-tuning rather than introduce a separate Python script (I did use a local Python script to generate these values), so the tuning logic stays in one place and doesn't drift between multiple implementations.

[nikwen]

Awesome, thank you! Doing that in a separate PR makes sense to me.

[nikwen]

Is this still true? Or are they derived from the output of that command by running another script?

[forforever73]

This is no longer accurate. I'll update the comment once the tooling is in place to reflect the actual generation workflow.

[forforever73]

@nikwen Thanks for the review! The CI failure should be unrelated to this PR. I've rebased onto the latest master, so let's see if it goes away after the rebase.

[nikwen]

Thanks for the updates and for working on the feature! :)

[nikwen]

@forforever73 Shall we merge this?

[ggerganov]

It should be good to merge, but we leave it to the author to decide.

[forforever73]

@nikwen Yes, please. I'll address the remaining items in follow-up pr.