Skip to content

llama : check that all the tensor data is in the model file #6885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Apr 25, 2024
Merged

llama : check that all the tensor data is in the model file #6885

merged 2 commits into from
Apr 25, 2024

Conversation

slaren
Copy link
Member

@slaren slaren commented Apr 24, 2024

Fixes #6774

@github-actions GitHub Actions
Copy link
Contributor

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 200 iterations 🚀

Expand details for performance related PR only
  • Concurrent users: 8, duration: 10m
  • HTTP request : avg=24209.78ms p(95)=42702.91ms fails=, finish reason: stop=80 truncated=120
  • Prompt processing (pp): avg=270.99tk/s p(95)=792.71tk/s
  • Token generation (tg): avg=19.05tk/s p(95)=24.62tk/s
  • ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=sl/check-mmap-bounds commit=98d9bea0fc548c3148b8006ac006a9a94b8fe3c1

prompt_tokens_seconds

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 200 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1713991272 --> 1713991912
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 252.03, 252.03, 252.03, 252.03, 252.03, 258.02, 258.02, 258.02, 258.02, 258.02, 282.99, 282.99, 282.99, 282.99, 282.99, 399.93, 399.93, 399.93, 399.93, 399.93, 422.57, 422.57, 422.57, 422.57, 422.57, 422.33, 422.33, 422.33, 422.33, 422.33, 424.38, 424.38, 424.38, 424.38, 424.38, 411.33, 411.33, 411.33, 411.33, 411.33, 424.71, 424.71, 424.71, 424.71, 424.71, 435.04, 435.04, 435.04, 435.04, 435.04, 445.84, 445.84, 445.84, 445.84, 445.84, 458.31, 458.31, 458.31, 458.31, 458.31, 479.7, 479.7, 479.7, 479.7, 479.7, 501.3, 501.3, 501.3, 501.3, 501.3, 504.07, 504.07, 504.07, 504.07, 504.07, 504.75, 504.75, 504.75, 504.75, 504.75, 505.4, 505.4, 505.4, 505.4, 505.4, 508.48, 508.48, 508.48, 508.48, 508.48, 509.29, 509.29, 509.29, 509.29, 509.29, 522.79, 522.79, 522.79, 522.79, 522.79, 523.25, 523.25, 523.25, 523.25, 523.25, 529.28, 529.28, 529.28, 529.28, 529.28, 533.79, 533.79, 533.79, 533.79, 533.79, 531.7, 531.7, 531.7, 531.7, 531.7, 545.72, 545.72, 545.72, 545.72, 545.72, 548.42, 548.42, 548.42, 548.42, 548.42, 549.42, 549.42, 549.42, 549.42, 549.42, 567.41, 567.41, 567.41, 567.41, 567.41, 567.52, 567.52, 567.52, 567.52, 567.52, 567.24, 567.24, 567.24, 567.24, 567.24, 567.0, 567.0, 567.0, 567.0, 567.0, 583.87, 583.87, 583.87, 583.87, 583.87, 589.86, 589.86, 589.86, 589.86, 589.86, 589.73, 589.73, 589.73, 589.73, 589.73, 590.56, 590.56, 590.56, 590.56, 590.56, 597.62, 597.62, 597.62, 597.62, 597.62, 601.21, 601.21, 601.21, 601.21, 601.21, 605.41, 605.41, 605.41, 605.41, 605.41, 606.8, 606.8, 606.8, 606.8, 606.8, 604.58, 604.58, 604.58, 604.58, 604.58, 603.01, 603.01, 603.01, 603.01, 603.01, 600.96, 600.96, 600.96, 600.96, 600.96, 601.48, 601.48, 601.48, 601.48, 601.48, 602.74, 602.74, 602.74, 602.74, 602.74, 604.87, 604.87, 604.87, 604.87, 604.87, 605.69, 605.69, 605.69, 605.69, 605.69, 606.6, 606.6, 606.6, 606.6, 606.6, 606.88, 606.88, 606.88, 606.88, 606.88, 606.33, 606.33, 606.33, 606.33, 606.33, 612.62, 612.62, 612.62, 612.62, 612.62, 617.83, 617.83, 617.83, 617.83, 617.83, 619.48, 619.48, 619.48, 619.48, 619.48, 619.17, 619.17, 619.17, 619.17, 619.17, 617.03, 617.03, 617.03, 617.03, 617.03, 616.9, 616.9, 616.9, 616.9, 616.9, 619.85, 619.85, 619.85, 619.85, 619.85, 620.67, 620.67, 620.67, 620.67, 620.67, 621.18, 621.18, 621.18, 621.18, 621.18, 625.12, 625.12, 625.12, 625.12, 625.12, 625.08, 625.08, 625.08, 625.08, 625.08, 625.08, 625.08, 625.08, 625.08, 625.08, 625.08]
Loading
predicted_tokens_seconds
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 200 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1713991272 --> 1713991912
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 28.99, 28.99, 28.99, 28.99, 28.99, 28.05, 28.05, 28.05, 28.05, 28.05, 25.53, 25.53, 25.53, 25.53, 25.53, 24.71, 24.71, 24.71, 24.71, 24.71, 23.39, 23.39, 23.39, 23.39, 23.39, 22.12, 22.12, 22.12, 22.12, 22.12, 20.75, 20.75, 20.75, 20.75, 20.75, 17.49, 17.49, 17.49, 17.49, 17.49, 17.32, 17.32, 17.32, 17.32, 17.32, 17.86, 17.86, 17.86, 17.86, 17.86, 18.25, 18.25, 18.25, 18.25, 18.25, 18.25, 18.25, 18.25, 18.25, 18.25, 18.63, 18.63, 18.63, 18.63, 18.63, 18.61, 18.61, 18.61, 18.61, 18.61, 18.62, 18.62, 18.62, 18.62, 18.62, 18.67, 18.67, 18.67, 18.67, 18.67, 18.87, 18.87, 18.87, 18.87, 18.87, 19.15, 19.15, 19.15, 19.15, 19.15, 19.32, 19.32, 19.32, 19.32, 19.32, 19.61, 19.61, 19.61, 19.61, 19.61, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.67, 19.64, 19.64, 19.64, 19.64, 19.64, 19.74, 19.74, 19.74, 19.74, 19.74, 19.76, 19.76, 19.76, 19.76, 19.76, 19.81, 19.81, 19.81, 19.81, 19.81, 19.96, 19.96, 19.96, 19.96, 19.96, 19.95, 19.95, 19.95, 19.95, 19.95, 19.93, 19.93, 19.93, 19.93, 19.93, 19.9, 19.9, 19.9, 19.9, 19.9, 19.77, 19.77, 19.77, 19.77, 19.77, 19.69, 19.69, 19.69, 19.69, 19.69, 19.5, 19.5, 19.5, 19.5, 19.5, 19.5, 19.5, 19.5, 19.5, 19.5, 19.15, 19.15, 19.15, 19.15, 19.15, 19.1, 19.1, 19.1, 19.1, 19.1, 19.05, 19.05, 19.05, 19.05, 19.05, 19.0, 19.0, 19.0, 19.0, 19.0, 18.75, 18.75, 18.75, 18.75, 18.75, 18.56, 18.56, 18.56, 18.56, 18.56, 18.38, 18.38, 18.38, 18.38, 18.38, 18.26, 18.26, 18.26, 18.26, 18.26, 17.93, 17.93, 17.93, 17.93, 17.93, 17.84, 17.84, 17.84, 17.84, 17.84, 17.82, 17.82, 17.82, 17.82, 17.82, 17.84, 17.84, 17.84, 17.84, 17.84, 17.85, 17.85, 17.85, 17.85, 17.85, 17.93, 17.93, 17.93, 17.93, 17.93, 17.96, 17.96, 17.96, 17.96, 17.96, 17.97, 17.97, 17.97, 17.97, 17.97, 17.94, 17.94, 17.94, 17.94, 17.94, 17.94, 17.94, 17.94, 17.94, 17.94, 17.79, 17.79, 17.79, 17.79, 17.79, 17.76, 17.76, 17.76, 17.76, 17.76, 17.71, 17.71, 17.71, 17.71, 17.71, 17.75, 17.75, 17.75, 17.75, 17.75, 17.79, 17.79, 17.79, 17.79, 17.79, 17.82, 17.82, 17.82, 17.82, 17.82, 17.87, 17.87, 17.87, 17.87, 17.87, 17.89, 17.89, 17.89, 17.89, 17.89, 18.07]
Loading

Details

kv_cache_usage_ratio

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 200 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1713991272 --> 1713991912
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.18, 0.18, 0.18, 0.18, 0.18, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.25, 0.34, 0.34, 0.34, 0.34, 0.34, 0.44, 0.44, 0.44, 0.44, 0.44, 0.48, 0.48, 0.48, 0.48, 0.48, 0.52, 0.52, 0.52, 0.52, 0.52, 0.21, 0.21, 0.21, 0.21, 0.21, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.2, 0.22, 0.22, 0.22, 0.22, 0.22, 0.21, 0.21, 0.21, 0.21, 0.21, 0.26, 0.26, 0.26, 0.26, 0.26, 0.23, 0.23, 0.23, 0.23, 0.23, 0.15, 0.15, 0.15, 0.15, 0.15, 0.21, 0.21, 0.21, 0.21, 0.21, 0.18, 0.18, 0.18, 0.18, 0.18, 0.14, 0.14, 0.14, 0.14, 0.14, 0.17, 0.17, 0.17, 0.17, 0.17, 0.21, 0.21, 0.21, 0.21, 0.21, 0.19, 0.19, 0.19, 0.19, 0.19, 0.27, 0.27, 0.27, 0.27, 0.27, 0.2, 0.2, 0.2, 0.2, 0.2, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23, 0.23, 0.23, 0.23, 0.23, 0.15, 0.15, 0.15, 0.15, 0.15, 0.18, 0.18, 0.18, 0.18, 0.18, 0.25, 0.25, 0.25, 0.25, 0.25, 0.21, 0.21, 0.21, 0.21, 0.21, 0.26, 0.26, 0.26, 0.26, 0.26, 0.25, 0.25, 0.25, 0.25, 0.25, 0.28, 0.28, 0.28, 0.28, 0.28, 0.29, 0.29, 0.29, 0.29, 0.29, 0.34, 0.34, 0.34, 0.34, 0.34, 0.4, 0.4, 0.4, 0.4, 0.4, 0.22, 0.22, 0.22, 0.22, 0.22, 0.27, 0.27, 0.27, 0.27, 0.27, 0.36, 0.36, 0.36, 0.36, 0.36, 0.45, 0.45, 0.45, 0.45, 0.45, 0.52, 0.52, 0.52, 0.52, 0.52, 0.48, 0.48, 0.48, 0.48, 0.48, 0.38, 0.38, 0.38, 0.38, 0.38, 0.35, 0.35, 0.35, 0.35, 0.35, 0.22, 0.22, 0.22, 0.22, 0.22, 0.24, 0.24, 0.24, 0.24, 0.24, 0.19, 0.19, 0.19, 0.19, 0.19, 0.22, 0.22, 0.22, 0.22, 0.22, 0.18, 0.18, 0.18, 0.18, 0.18, 0.21, 0.21, 0.21, 0.21, 0.21, 0.2, 0.2, 0.2, 0.2, 0.2, 0.3, 0.3, 0.3, 0.3, 0.3, 0.38, 0.38, 0.38, 0.38, 0.38, 0.41, 0.41, 0.41, 0.41, 0.41, 0.27, 0.27, 0.27, 0.27, 0.27, 0.26, 0.26, 0.26, 0.26, 0.26, 0.18, 0.18, 0.18, 0.18, 0.18, 0.12, 0.12, 0.12, 0.12, 0.12, 0.19, 0.19, 0.19, 0.19, 0.19, 0.25, 0.25, 0.25, 0.25, 0.25, 0.13, 0.13, 0.13, 0.13, 0.13, 0.22, 0.22, 0.22, 0.22, 0.22, 0.28]
Loading
requests_processing
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 200 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1713991272 --> 1713991912
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 0.0]
Loading

@slaren slaren merged commit 0ead1f1 into master Apr 25, 2024
@slaren slaren deleted the sl/check-mmap-bounds branch April 25, 2024 13:23
@cebtenzzre cebtenzzre mentioned this pull request Apr 25, 2024
Closed
@phymbert
Copy link
Collaborator

I am facing blk.3.ffn_up.weight data is not within the file bounds. I am confused because it was working fine previously.

There is a chance some shard were not well downloaded, because I have seen failed: partially download, but normally if .etag and lastModified are present, the download has succeeded.

I will add more info in the error message to understand which file is corrupted.

@phymbert
Copy link
Collaborator

I will add more info in the error message to understand which file is corrupted.

NFS issue... Nice check <3

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ggerganov ggerganov ggerganov approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Truncated model files can cause llama.cpp to crash when using mmap
3 participants
@slaren @phymbert @ggerganov