Skip to content

[gemma3] fix bidirectional attention mask #38080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
May 20, 2025
Merged

[gemma3] fix bidirectional attention mask #38080

merged 5 commits into from
May 20, 2025

Conversation

@zucchini-nlp
Copy link
Member

@zucchini-nlp zucchini-nlp commented May 12, 2025

What does this PR do?

Fixes #38053. Images cannot attend to future images but in current implementation we unmask attention within all images

@github-actions github-actions bot marked this pull request as draft May 12, 2025 11:05
@github-actions
Copy link
Contributor

github-actions bot commented May 12, 2025

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

@zucchini-nlp zucchini-nlp marked this pull request as ready for review May 12, 2025 11:05
@zucchini-nlp
Copy link
Member Author

zucchini-nlp commented May 12, 2025

[run-slow] gemma3

@zucchini-nlp
Copy link
Member Author

zucchini-nlp commented May 12, 2025

run-slow: gemma3

@github-actions
Copy link
Contributor

github-actions bot commented May 12, 2025

This comment contains run-slow, running the specified jobs:

models: ['models/gemma3']
quantizations: [] ...

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented May 12, 2025

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp zucchini-nlp requested a review from ArthurZucker May 12, 2025 11:30
@sammysun0711 sammysun0711 mentioned this pull request May 14, 2025
Open
ArthurZucker
ArthurZucker reviewed May 20, 2025
View reviewed changes
@zucchini-nlp
Copy link
Member Author

zucchini-nlp commented May 20, 2025

image

What I got with the current state, and a few more tweaks for gemma3 locally (for the weird image token 🙃 )

ArthurZucker
ArthurZucker approved these changes May 20, 2025
View reviewed changes
Copy link
Collaborator

@ArthurZucker ArthurZucker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks it looks better!

@zucchini-nlp zucchini-nlp merged commit f834d36 into huggingface:main May 20, 2025
14 checks passed
faaany pushed a commit to faaany/transformers that referenced this pull request May 21, 2025
@zucchini-nlp @faaany
[gemma3] fix bidirectional attention mask (huggingface#38080)
0175c4f 
* fix attn mask

* attn viz doesn't show yello cubes between images

* bucketize made it hard with different number of crops

* fixup
xvyv99 pushed a commit to xvyv99/transformers that referenced this pull request May 21, 2025
@zucchini-nlp @xvyv99
[gemma3] fix bidirectional attention mask (huggingface#38080)
22c3fb7 
* fix attn mask

* attn viz doesn't show yello cubes between images

* bucketize made it hard with different number of crops

* fixup
@Cyrilvallez Cyrilvallez mentioned this pull request May 22, 2025
Merged
@brb-nv brb-nv mentioned this pull request Jul 13, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ArthurZucker ArthurZucker ArthurZucker approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Attention mask for multi-image input in gemma3

3 participants

@zucchini-nlp @HuggingFaceDocBuilderDev @ArthurZucker