Skip to content

Add revised benchmarking logic and results #9

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
May 28, 2024
Merged

Add revised benchmarking logic and results #9

merged 6 commits into from
May 28, 2024

Conversation

jacobthebanana
Copy link
Collaborator

New features added in this pull request:

  • benchmark.py now skips eval when benchmarking training throughput to ensure a consistent measure of throughput.
  • When the experiment includes different batch sizes on the same hardware configuration, parse_benchmark.py would now report the batch size that maximizes median throughput for this model given the context window.

@jacobthebanana
Revised estimation of batch count, directly retrieving from len(train…
573b6f7 
…_dataloader).

Deleted unused timer_handle argument in Trainer.
Revised handling of "max_seq_len" override in benchmarking.
Added support for automatic switching between  lora and full-rank sharding scheme in benchmarking.
@jacobthebanana
Revised handling of unspecified max_seq_length.
dfbd388 
Added llama-3 to benchmark model_list.
@jacobthebanana
Benchmarking: Revised benchmark script to ensure consistent per-devic…
555c55a 
…e train batch size.
@jacobthebanana
Benchmarking: replaced trainer.step with trainer.train_step to avoid …
d6e3355 
…eval overhead in benchmarking.

Revised benchmark parsing logic; display optimal batch size for each context width value.
@jacobthebanana
Benchmarking: Updated reference throughput based on updated logic.
0703e07
@jacobthebanana jacobthebanana requested a review from adil-a May 9, 2024 15:06
adil-a
adil-a approved these changes May 28, 2024
View reviewed changes
Copy link
Collaborator

@adil-a adil-a left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@adil-a adil-a merged commit 9045f08 into master May 28, 2024
@adil-a adil-a deleted the jjt/lora-benchmarking-revisions branch May 28, 2024 15:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adil-a adil-a adil-a approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jacobthebanana @adil-a