server: fix reported top tokens for temperature 0

[JohannesGaessler]

Fixes #7197 .

I had incorrectly assumed that with greedy sampling the size of the llama_token_data_array would be 1. Instead it is the vocabulary size and I did not catch this during testing. There is also the issue that the tokens are not sorted at all which would cause issues when requesting only the probability of the top token with temperature 0. Edit: ... when keeping the current interpretation of n_considered.

This PR renames n_considered to n_valid and slightly changes the intended interpretation. It now represents the number of correct top tokens with correct probabilities. For temperature 0 this is set to 0 so that the server always fetches the top tokens.

[github-actions]

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 541 iterations 🚀

Expand details for performance related PR only

• Concurrent users: 8, duration: 10m
• HTTP request : avg=8627.45ms p(95)=22197.77ms fails=, finish reason: stop=480 truncated=61
• Prompt processing (pp): avg=99.76tk/s p(95)=479.34tk/s
• Token generation (tg): avg=32.01tk/s p(95)=45.91tk/s
• ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=sampling-fix-n_cons commit=fe47a6dcba11ed52f653419376411285e7fbb597

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1715423012 --> 1715423642
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 775.21, 775.21, 775.21, 775.21, 775.21, 711.38, 711.38, 711.38, 711.38, 711.38, 726.97, 726.97, 726.97, 726.97, 726.97, 756.06, 756.06, 756.06, 756.06, 756.06, 812.23, 812.23, 812.23, 812.23, 812.23, 812.27, 812.27, 812.27, 812.27, 812.27, 811.21, 811.21, 811.21, 811.21, 811.21, 812.37, 812.37, 812.37, 812.37, 812.37, 818.22, 818.22, 818.22, 818.22, 818.22, 818.33, 818.33, 818.33, 818.33, 818.33, 838.61, 838.61, 838.61, 838.61, 838.61, 851.74, 851.74, 851.74, 851.74, 851.74, 872.73, 872.73, 872.73, 872.73, 872.73, 852.59, 852.59, 852.59, 852.59, 852.59, 843.56, 843.56, 843.56, 843.56, 843.56, 843.7, 843.7, 843.7, 843.7, 843.7, 840.41, 840.41, 840.41, 840.41, 840.41, 840.11, 840.11, 840.11, 840.11, 840.11, 838.99, 838.99, 838.99, 838.99, 838.99, 843.82, 843.82, 843.82, 843.82, 843.82, 842.96, 842.96, 842.96, 842.96, 842.96, 848.15, 848.15, 848.15, 848.15, 848.15, 861.49, 861.49, 861.49, 861.49, 861.49, 860.39, 860.39, 860.39, 860.39, 860.39, 857.0, 857.0, 857.0, 857.0, 857.0, 850.88, 850.88, 850.88, 850.88, 850.88, 847.25, 847.25, 847.25, 847.25, 847.25, 845.72, 845.72, 845.72, 845.72, 845.72, 845.57, 845.57, 845.57, 845.57, 845.57, 849.34, 849.34, 849.34, 849.34, 849.34, 850.28, 850.28, 850.28, 850.28, 850.28, 850.57, 850.57, 850.57, 850.57, 850.57, 849.74, 849.74, 849.74, 849.74, 849.74, 854.35, 854.35, 854.35, 854.35, 854.35, 860.63, 860.63, 860.63, 860.63, 860.63, 864.38, 864.38, 864.38, 864.38, 864.38, 860.25, 860.25, 860.25, 860.25, 860.25, 857.79, 857.79, 857.79, 857.79, 857.79, 859.6, 859.6, 859.6, 859.6, 859.6, 862.7, 862.7, 862.7, 862.7, 862.7, 863.24, 863.24, 863.24, 863.24, 863.24, 847.68, 847.68, 847.68, 847.68, 847.68, 829.8, 829.8, 829.8, 829.8, 829.8, 829.21, 829.21, 829.21, 829.21, 829.21, 827.62, 827.62, 827.62, 827.62, 827.62, 827.65, 827.65, 827.65, 827.65, 827.65, 833.7, 833.7, 833.7, 833.7, 833.7, 833.2, 833.2, 833.2, 833.2, 833.2, 838.97, 838.97, 838.97, 838.97, 838.97, 837.64, 837.64, 837.64, 837.64, 837.64, 840.76, 840.76, 840.76, 840.76, 840.76, 840.58, 840.58, 840.58, 840.58, 840.58, 838.95, 838.95, 838.95, 838.95, 838.95, 845.5, 845.5, 845.5, 845.5, 845.5, 845.06, 845.06, 845.06, 845.06, 845.06, 846.02, 846.02, 846.02, 846.02, 846.02, 845.45, 845.45, 845.45, 845.45, 845.45, 846.66, 846.66, 846.66, 846.66, 846.66, 847.76, 847.76, 847.76, 847.76, 847.76, 848.64, 848.64, 848.64, 848.64, 848.64, 848.64]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1715423012 --> 1715423642
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 41.67, 41.67, 41.67, 41.67, 41.67, 33.26, 33.26, 33.26, 33.26, 33.26, 29.91, 29.91, 29.91, 29.91, 29.91, 30.37, 30.37, 30.37, 30.37, 30.37, 31.15, 31.15, 31.15, 31.15, 31.15, 32.37, 32.37, 32.37, 32.37, 32.37, 33.65, 33.65, 33.65, 33.65, 33.65, 33.89, 33.89, 33.89, 33.89, 33.89, 33.92, 33.92, 33.92, 33.92, 33.92, 33.91, 33.91, 33.91, 33.91, 33.91, 33.72, 33.72, 33.72, 33.72, 33.72, 33.33, 33.33, 33.33, 33.33, 33.33, 32.65, 32.65, 32.65, 32.65, 32.65, 32.39, 32.39, 32.39, 32.39, 32.39, 31.04, 31.04, 31.04, 31.04, 31.04, 31.37, 31.37, 31.37, 31.37, 31.37, 31.74, 31.74, 31.74, 31.74, 31.74, 31.81, 31.81, 31.81, 31.81, 31.81, 31.61, 31.61, 31.61, 31.61, 31.61, 31.33, 31.33, 31.33, 31.33, 31.33, 31.02, 31.02, 31.02, 31.02, 31.02, 31.2, 31.2, 31.2, 31.2, 31.2, 31.24, 31.24, 31.24, 31.24, 31.24, 31.03, 31.03, 31.03, 31.03, 31.03, 31.08, 31.08, 31.08, 31.08, 31.08, 31.19, 31.19, 31.19, 31.19, 31.19, 30.94, 30.94, 30.94, 30.94, 30.94, 30.65, 30.65, 30.65, 30.65, 30.65, 30.55, 30.55, 30.55, 30.55, 30.55, 30.67, 30.67, 30.67, 30.67, 30.67, 30.82, 30.82, 30.82, 30.82, 30.82, 31.02, 31.02, 31.02, 31.02, 31.02, 31.15, 31.15, 31.15, 31.15, 31.15, 31.11, 31.11, 31.11, 31.11, 31.11, 30.94, 30.94, 30.94, 30.94, 30.94, 30.81, 30.81, 30.81, 30.81, 30.81, 30.67, 30.67, 30.67, 30.67, 30.67, 30.7, 30.7, 30.7, 30.7, 30.7, 30.8, 30.8, 30.8, 30.8, 30.8, 30.96, 30.96, 30.96, 30.96, 30.96, 31.11, 31.11, 31.11, 31.11, 31.11, 31.11, 31.11, 31.11, 31.11, 31.11, 30.86, 30.86, 30.86, 30.86, 30.86, 30.47, 30.47, 30.47, 30.47, 30.47, 29.92, 29.92, 29.92, 29.92, 29.92, 29.4, 29.4, 29.4, 29.4, 29.4, 29.34, 29.34, 29.34, 29.34, 29.34, 29.34, 29.34, 29.34, 29.34, 29.34, 29.5, 29.5, 29.5, 29.5, 29.5, 29.54, 29.54, 29.54, 29.54, 29.54, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.68, 29.57, 29.57, 29.57, 29.57, 29.57, 29.5, 29.5, 29.5, 29.5, 29.5, 29.58, 29.58, 29.58, 29.58, 29.58, 29.76, 29.76, 29.76, 29.76, 29.76, 29.8, 29.8, 29.8, 29.8, 29.8, 29.89, 29.89, 29.89, 29.89, 29.89, 29.98, 29.98, 29.98, 29.98, 29.98, 30.04]
                    

Loading

Details

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1715423012 --> 1715423642
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.48, 0.48, 0.48, 0.48, 0.48, 0.38, 0.38, 0.38, 0.38, 0.38, 0.28, 0.28, 0.28, 0.28, 0.28, 0.16, 0.16, 0.16, 0.16, 0.16, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.14, 0.14, 0.14, 0.14, 0.14, 0.19, 0.19, 0.19, 0.19, 0.19, 0.25, 0.25, 0.25, 0.25, 0.25, 0.19, 0.19, 0.19, 0.19, 0.19, 0.31, 0.31, 0.31, 0.31, 0.31, 0.19, 0.19, 0.19, 0.19, 0.19, 0.17, 0.17, 0.17, 0.17, 0.17, 0.38, 0.38, 0.38, 0.38, 0.38, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.13, 0.13, 0.13, 0.13, 0.13, 0.22, 0.22, 0.22, 0.22, 0.22, 0.25, 0.25, 0.25, 0.25, 0.25, 0.21, 0.21, 0.21, 0.21, 0.21, 0.13, 0.13, 0.13, 0.13, 0.13, 0.16, 0.16, 0.16, 0.16, 0.16, 0.23, 0.23, 0.23, 0.23, 0.23, 0.09, 0.09, 0.09, 0.09, 0.09, 0.15, 0.15, 0.15, 0.15, 0.15, 0.31, 0.31, 0.31, 0.31, 0.31, 0.35, 0.35, 0.35, 0.35, 0.35, 0.21, 0.21, 0.21, 0.21, 0.21, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.14, 0.14, 0.14, 0.14, 0.09, 0.09, 0.09, 0.09, 0.09, 0.13, 0.13, 0.13, 0.13, 0.13, 0.14, 0.14, 0.14, 0.14, 0.14, 0.22, 0.22, 0.22, 0.22, 0.22, 0.19, 0.19, 0.19, 0.19, 0.19, 0.32, 0.32, 0.32, 0.32, 0.32, 0.17, 0.17, 0.17, 0.17, 0.17, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.1, 0.1, 0.1, 0.1, 0.1, 0.16, 0.16, 0.16, 0.16, 0.16, 0.46, 0.46, 0.46, 0.46, 0.46, 0.5, 0.5, 0.5, 0.5, 0.5, 0.59, 0.59, 0.59, 0.59, 0.59, 0.39, 0.39, 0.39, 0.39, 0.39, 0.14, 0.14, 0.14, 0.14, 0.14, 0.23, 0.23, 0.23, 0.23, 0.23, 0.11, 0.11, 0.11, 0.11, 0.11, 0.24, 0.24, 0.24, 0.24, 0.24, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.28, 0.28, 0.28, 0.28, 0.28, 0.24, 0.24, 0.24, 0.24, 0.24, 0.25, 0.25, 0.25, 0.25, 0.25, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.1, 0.1, 0.1, 0.1, 0.1, 0.13, 0.13, 0.13, 0.13, 0.13, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18]
                    

Loading

More

---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 541 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1715423012 --> 1715423642
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 7.0, 7.0, 7.0, 7.0, 7.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 2.0, 2.0, 2.0, 2.0, 2.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 1.0]
                    

Loading