ggml : generalize GGML_OP_CONCAT

[ggerganov]

Support concat across all dimensions:

https://pytorch.org/docs/stable/generated/torch.cat.html#torch.cat

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 540 iterations 🚀

Expand details for performance related PR only

• Concurrent users: 8, duration: 10m
• HTTP request : avg=8676.14ms p(95)=22491.83ms fails=, finish reason: stop=483 truncated=57
• Prompt processing (pp): avg=94.05tk/s p(95)=381.08tk/s
• Token generation (tg): avg=45.19tk/s p(95)=45.12tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=gg/concat commit=df80e03899bfcb0e2213231dc4d68ca4bd56ce78

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 301.67, 301.67, 301.67, 301.67, 301.67, 738.48, 738.48, 738.48, 738.48, 738.48, 764.74, 764.74, 764.74, 764.74, 764.74, 780.4, 780.4, 780.4, 780.4, 780.4, 806.59, 806.59, 806.59, 806.59, 806.59, 864.43, 864.43, 864.43, 864.43, 864.43, 850.69, 850.69, 850.69, 850.69, 850.69, 863.45, 863.45, 863.45, 863.45, 863.45, 873.71, 873.71, 873.71, 873.71, 873.71, 886.12, 886.12, 886.12, 886.12, 886.12, 883.73, 883.73, 883.73, 883.73, 883.73, 896.91, 896.91, 896.91, 896.91, 896.91, 862.68, 862.68, 862.68, 862.68, 862.68, 839.32, 839.32, 839.32, 839.32, 839.32, 858.3, 858.3, 858.3, 858.3, 858.3, 865.65, 865.65, 865.65, 865.65, 865.65, 866.63, 866.63, 866.63, 866.63, 866.63, 871.33, 871.33, 871.33, 871.33, 871.33, 866.91, 866.91, 866.91, 866.91, 866.91, 886.88, 886.88, 886.88, 886.88, 886.88, 886.75, 886.75, 886.75, 886.75, 886.75, 884.16, 884.16, 884.16, 884.16, 884.16, 889.92, 889.92, 889.92, 889.92, 889.92, 891.06, 891.06, 891.06, 891.06, 891.06, 886.79, 886.79, 886.79, 886.79, 886.79, 875.56, 875.56, 875.56, 875.56, 875.56, 876.88, 876.88, 876.88, 876.88, 876.88, 870.0, 870.0, 870.0, 870.0, 870.0, 864.36, 864.36, 864.36, 864.36, 864.36, 861.57, 861.57, 861.57, 861.57, 861.57, 863.58, 863.58, 863.58, 863.58, 863.58, 865.83, 865.83, 865.83, 865.83, 865.83, 863.91, 863.91, 863.91, 863.91, 863.91, 864.36, 864.36, 864.36, 864.36, 864.36, 862.52, 862.52, 862.52, 862.52, 862.52, 863.72, 863.72, 863.72, 863.72, 863.72, 861.72, 861.72, 861.72, 861.72, 861.72, 864.95, 864.95, 864.95, 864.95, 864.95, 855.62, 855.62, 855.62, 855.62, 855.62, 853.63, 853.63, 853.63, 853.63, 853.63, 857.09, 857.09, 857.09, 857.09, 857.09, 860.13, 860.13, 860.13, 860.13, 860.13, 866.95, 866.95, 866.95, 866.95, 866.95, 863.39, 863.39, 863.39, 863.39, 863.39, 854.45, 854.45, 854.45, 854.45, 854.45, 853.86, 853.86, 853.86, 853.86, 853.86, 852.42, 852.42, 852.42, 852.42, 852.42, 855.99, 855.99, 855.99, 855.99, 855.99, 857.7, 857.7, 857.7, 857.7, 857.7, 856.48, 856.48, 856.48, 856.48, 856.48, 862.56, 862.56, 862.56, 862.56, 862.56, 861.35, 861.35, 861.35, 861.35, 861.35, 862.49, 862.49, 862.49, 862.49, 862.49, 865.75, 865.75, 865.75, 865.75, 865.75, 866.97, 866.97, 866.97, 866.97, 866.97, 869.68, 869.68, 869.68, 869.68, 869.68, 870.03, 870.03, 870.03, 870.03, 870.03, 869.5, 869.5, 869.5, 869.5, 869.5, 870.19, 870.19, 870.19, 870.19, 870.19, 871.56, 871.56, 871.56, 871.56, 871.56, 872.32, 872.32, 872.32, 872.32, 872.32, 872.32]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 34.23, 34.23, 34.23, 34.23, 34.23, 38.46, 38.46, 38.46, 38.46, 38.46, 25.94, 25.94, 25.94, 25.94, 25.94, 27.73, 27.73, 27.73, 27.73, 27.73, 28.67, 28.67, 28.67, 28.67, 28.67, 29.35, 29.35, 29.35, 29.35, 29.35, 30.17, 30.17, 30.17, 30.17, 30.17, 30.94, 30.94, 30.94, 30.94, 30.94, 31.85, 31.85, 31.85, 31.85, 31.85, 32.04, 32.04, 32.04, 32.04, 32.04, 32.2, 32.2, 32.2, 32.2, 32.2, 32.24, 32.24, 32.24, 32.24, 32.24, 32.3, 32.3, 32.3, 32.3, 32.3, 31.48, 31.48, 31.48, 31.48, 31.48, 31.32, 31.32, 31.32, 31.32, 31.32, 30.42, 30.42, 30.42, 30.42, 30.42, 28.87, 28.87, 28.87, 28.87, 28.87, 28.32, 28.32, 28.32, 28.32, 28.32, 28.52, 28.52, 28.52, 28.52, 28.52, 28.57, 28.57, 28.57, 28.57, 28.57, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.47, 28.55, 28.55, 28.55, 28.55, 28.55, 28.85, 28.85, 28.85, 28.85, 28.85, 29.15, 29.15, 29.15, 29.15, 29.15, 29.1, 29.1, 29.1, 29.1, 29.1, 29.32, 29.32, 29.32, 29.32, 29.32, 29.52, 29.52, 29.52, 29.52, 29.52, 29.35, 29.35, 29.35, 29.35, 29.35, 29.43, 29.43, 29.43, 29.43, 29.43, 29.65, 29.65, 29.65, 29.65, 29.65, 29.78, 29.78, 29.78, 29.78, 29.78, 29.99, 29.99, 29.99, 29.99, 29.99, 30.06, 30.06, 30.06, 30.06, 30.06, 30.25, 30.25, 30.25, 30.25, 30.25, 30.16, 30.16, 30.16, 30.16, 30.16, 30.11, 30.11, 30.11, 30.11, 30.11, 30.0, 30.0, 30.0, 30.0, 30.0, 29.53, 29.53, 29.53, 29.53, 29.53, 29.45, 29.45, 29.45, 29.45, 29.45, 29.65, 29.65, 29.65, 29.65, 29.65, 29.74, 29.74, 29.74, 29.74, 29.74, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.84, 29.36, 29.36, 29.36, 29.36, 29.36, 29.15, 29.15, 29.15, 29.15, 29.15, 28.51, 28.51, 28.51, 28.51, 28.51, 28.46, 28.46, 28.46, 28.46, 28.46, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.42, 28.59, 28.59, 28.59, 28.59, 28.59, 28.61, 28.61, 28.61, 28.61, 28.61, 28.72, 28.72, 28.72, 28.72, 28.72, 28.73, 28.73, 28.73, 28.73, 28.73, 28.66, 28.66, 28.66, 28.66, 28.66, 28.57, 28.57, 28.57, 28.57, 28.57, 28.64, 28.64, 28.64, 28.64, 28.64, 28.74, 28.74, 28.74, 28.74, 28.74, 28.84, 28.84, 28.84, 28.84, 28.84, 28.92, 28.92, 28.92, 28.92, 28.92, 29.02, 29.02, 29.02, 29.02, 29.02, 29.11]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.08, 0.08, 0.08, 0.08, 0.08, 0.36, 0.36, 0.36, 0.36, 0.36, 0.26, 0.26, 0.26, 0.26, 0.26, 0.22, 0.22, 0.22, 0.22, 0.22, 0.14, 0.14, 0.14, 0.14, 0.14, 0.23, 0.23, 0.23, 0.23, 0.23, 0.22, 0.22, 0.22, 0.22, 0.22, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.16, 0.16, 0.19, 0.19, 0.19, 0.19, 0.19, 0.23, 0.23, 0.23, 0.23, 0.23, 0.19, 0.19, 0.19, 0.19, 0.19, 0.27, 0.27, 0.27, 0.27, 0.27, 0.3, 0.3, 0.3, 0.3, 0.3, 0.45, 0.45, 0.45, 0.45, 0.45, 0.39, 0.39, 0.39, 0.39, 0.39, 0.31, 0.31, 0.31, 0.31, 0.31, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.25, 0.25, 0.25, 0.25, 0.25, 0.22, 0.22, 0.22, 0.22, 0.22, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.3, 0.3, 0.3, 0.3, 0.3, 0.1, 0.1, 0.1, 0.1, 0.1, 0.16, 0.16, 0.16, 0.16, 0.16, 0.33, 0.33, 0.33, 0.33, 0.33, 0.21, 0.21, 0.21, 0.21, 0.21, 0.12, 0.12, 0.12, 0.12, 0.12, 0.16, 0.16, 0.16, 0.16, 0.16, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.16, 0.32, 0.32, 0.32, 0.32, 0.32, 0.28, 0.28, 0.28, 0.28, 0.28, 0.48, 0.48, 0.48, 0.48, 0.48, 0.24, 0.24, 0.24, 0.24, 0.24, 0.07, 0.07, 0.07, 0.07, 0.07, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.31, 0.31, 0.31, 0.31, 0.31, 0.42, 0.42, 0.42, 0.42, 0.42, 0.45, 0.45, 0.45, 0.45, 0.45, 0.39, 0.39, 0.39, 0.39, 0.39, 0.33, 0.33, 0.33, 0.33, 0.33, 0.26, 0.26, 0.26, 0.26, 0.26, 0.23, 0.23, 0.23, 0.23, 0.23, 0.11, 0.11, 0.11, 0.11, 0.11, 0.21, 0.21, 0.21, 0.21, 0.21, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.29, 0.29, 0.29, 0.29, 0.29, 0.15, 0.15, 0.15, 0.15, 0.15, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23, 0.23, 0.21, 0.21, 0.21, 0.21, 0.21, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 540 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1716882871 --> 1716883501
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 1.0, 1.0, 1.0, 1.0, 1.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 3.0, 3.0, 3.0, 3.0, 3.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0, 1.0, 1.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 1.0]
                    

Loading

[ggerganov]

The CUDA implementation probably should be improved - I tried to keep the original logic, but I think it would be better to have a single kernel accepting dim as argument

[ggerganov]

I realized there was a bug in yesterday's CPU/Metal implementation. Should be good now