Skip to content

Attention Sinks Perf Boost #78

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 25 commits into from
Aug 9, 2025
Merged

Attention Sinks Perf Boost #78

merged 25 commits into from
Aug 9, 2025

Conversation

LucasWilkinson
Copy link
Collaborator

@LucasWilkinson LucasWilkinson commented Aug 7, 2025 •
edited
Loading

Shout-out to @jayhshah (the performance wizard 🪄) for the implementation

PR

reasoning-effort: low
model: openai/gpt-oss-20b

Results (n=5)
mean_chars: 53.7662878788
stderr_chars: 1.6426099883
mean_score: 0.5571969697
stderr_score: 0.0072987006


Main

reasoning-effort: low
model: openai/gpt-oss-20b

Results (n=5)
mean_chars: 53.2223484848
stderr_chars: 1.8772311021
mean_score: 0.5637626263
stderr_score: 0.0020261119

@LucasWilkinson LucasWilkinson mentioned this pull request Aug 7, 2025
Closed
jayhshah
jayhshah reviewed Aug 7, 2025
View reviewed changes
@jayhshah jayhshah force-pushed the lwilkinson/aux-fast-api-clean branch from 994e966 to deb7484 Compare August 8, 2025 00:46
@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/aux-fast-api-clean branch from deb7484 to e1f506c Compare August 8, 2025 01:07
jayhshah and others added 22 commits August 8, 2025 01:08
@jayhshah @LucasWilkinson
add sink test to attention_ref
18a0cae 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
embed sink test into main flash attn test script
684c203 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
change local block positioning for fast path
49be39d 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
add tma gqa modifications
bf2ad39 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix exploding build
07b6e77 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
try to fix exploding build again
3f8cfbe 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
prune other hdims to keep build stable
8d4de3c 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
tweak tile size for causal not local case
2145fea 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
split compilation
f293967 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix error with varlen q
95cc109 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
split compilation for root setup
e1217ab 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
renable use one mma wg
1a3cd09 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
update for hdim diff
5d58d4b 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
include comments on how to enable hdim diff in cmakelists
6a2c39a 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix test variable
0529f4f 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
update pack gqa heuristic
a4e9b0f 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
fix comment
53b576f 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
split disable hdim diff macro into 64 and 192, enable 192 by default …
a763abf 
…in cmakelists

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
add assert to check using Hopper kernels with s_aux
786873d 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@jayhshah @LucasWilkinson
more logical placement of assert checks for hdim diff in flash api
e5935eb 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
cherrypick
323bb43 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson
typo fix
4bca7a3 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson force-pushed the lwilkinson/aux-fast-api-clean branch from e1f506c to 4bca7a3 Compare August 8, 2025 01:08
@LucasWilkinson
review comments
330171b 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson mentioned this pull request Aug 8, 2025
Open
4 tasks
jayhshah
jayhshah reviewed Aug 8, 2025
View reviewed changes
@LucasWilkinson
review comment
4e44ba2 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson changed the title [WIP] Attention Sinks Perf Boost Attention Sinks Perf Boost Aug 9, 2025
@LucasWilkinson LucasWilkinson marked this pull request as ready for review August 9, 2025 03:58
@LucasWilkinson
Merge remote-tracking branch 'upstream/main' into lwilkinson/aux-fast…
6a3d9c8 
…-api-clean

Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
@LucasWilkinson LucasWilkinson merged commit 2d3b750 into main Aug 9, 2025
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jayhshah jayhshah jayhshah left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@LucasWilkinson @jayhshah