Skip to content

Finish fixes for padded dataloader #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
lhallee merged 5 commits into from
Jan 3, 2025
Merged

Finish fixes for padded dataloader #8

lhallee merged 5 commits into from
Jan 3, 2025

Conversation

@lapp0
Copy link
Collaborator

@lapp0 lapp0 commented Jan 2, 2025 •
edited
Loading

See #5 (comment)

  • don't skip first sample
  • average val / test loss by token count (don't change how gradients are averaged though)
  • don't add 1 to seq len (artifact from causal lm loss)

@lhallee lhallee merged commit d94f7c6 into master Jan 3, 2025
@lhallee lhallee deleted the padded-dataloader branch June 20, 2025 19:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@lapp0 @lhallee