Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CUDA: fix broken oob check for FA vec f32 kernel #7904

Merged
merged 1 commit into from
Jun 12, 2024
Merged

CUDA: fix broken oob check for FA vec f32 kernel #7904

merged 1 commit into from
Jun 12, 2024

Conversation

JohannesGaessler
Copy link
Collaborator

Fixes #7903 .

The problem was a typo in the FP32 FlashAttention vec kernel. Instead of ic0 + j < ne01 the check for out-of-bounds reads was ic0 + j.

@JohannesGaessler JohannesGaessler mentioned this pull request Jun 12, 2024
Closed
@JohannesGaessler JohannesGaessler merged commit 9635529 into ggerganov:master Jun 12, 2024
55 of 60 checks passed
@github-actions github-actions bot added Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Jun 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@slaren slaren slaren approved these changes

Assignees
No one assigned
Labels
ggml changes relating to the ggml tensor library for machine learning Nvidia GPU Issues specific to Nvidia GPUs
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Bug: flash attention produces corrupt output with batched generation
2 participants
@JohannesGaessler @slaren