Skip to content

Conversation

@yukirora
Copy link
Contributor

@yukirora yukirora commented Sep 5, 2025

Description
add option to exclude data copy time in model benchmarks.

Major Revision

  • add an option --exclude_copy_time
  • move start time after data copy finish in measured time

@yukirora yukirora added the benchmarks SuperBench Benchmarks label Sep 5, 2025
@yukirora yukirora requested a review from a team as a code owner September 5, 2025 07:59
@codecov
Copy link

codecov bot commented Sep 9, 2025

Codecov Report

❌ Patch coverage is 52.00000% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.97%. Comparing base (ad8e014) to head (761b973).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...rbench/benchmarks/model_benchmarks/pytorch_bert.py 50.00% 2 Missing ⚠️
...erbench/benchmarks/model_benchmarks/pytorch_cnn.py 50.00% 2 Missing ⚠️
...rbench/benchmarks/model_benchmarks/pytorch_gpt2.py 50.00% 2 Missing ⚠️
...bench/benchmarks/model_benchmarks/pytorch_llama.py 50.00% 2 Missing ⚠️
...rbench/benchmarks/model_benchmarks/pytorch_lstm.py 50.00% 2 Missing ⚠️
...enchmarks/model_benchmarks/pytorch_mixtral_impl.py 50.00% 2 Missing ⚠️

❌ Your patch status has failed because the patch coverage (52.00%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #734      +/-   ##
==========================================
- Coverage   86.08%   85.97%   -0.12%     
==========================================
  Files         102      102              
  Lines        7553     7578      +25     
==========================================
+ Hits         6502     6515      +13     
- Misses       1051     1063      +12     
Flag Coverage Δ
cpu-python3.10-unit-test 71.05% <20.00%> (-0.18%) ⬇️
cpu-python3.12-unit-test 71.05% <20.00%> (-0.18%) ⬇️
cpu-python3.7-unit-test 70.49% <20.00%> (-0.17%) ⬇️
cuda-unit-test 83.86% <52.00%> (-0.11%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@cp5555 cp5555 added the model-benchmarks Model Benchmark Test for SuperBench Benchmarks label Sep 18, 2025
@yukirora yukirora enabled auto-merge (squash) September 19, 2025 02:05
@abuccts abuccts requested a review from Copilot September 22, 2025 08:44
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds a new --no_copy option to model benchmarks that allows excluding data copy time from benchmark measurements. When enabled, the timing measurement starts after the GPU data copy operation is completed rather than before it.

  • Added a new command-line argument --no_copy to exclude data copy time from benchmarks
  • Modified timing logic in all model benchmark implementations to optionally start timing after GPU memory transfers
  • Updated test documentation to reflect the new option

Reviewed Changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

File Description
model_base.py Adds the new --no_copy command-line argument definition
pytorch_*.py (6 files) Implements the timing logic to start measurement after GPU data copy when --no_copy is enabled
test_model_base.py Updates test documentation to include the new --no_copy option

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

@yukirora yukirora merged commit 76066b6 into main Sep 29, 2025
23 of 25 checks passed
@yukirora yukirora deleted the yutji/model-nodatacp branch September 29, 2025 07:24
@guoshzhao guoshzhao mentioned this pull request Oct 2, 2025
30 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmarks SuperBench Benchmarks model-benchmarks Model Benchmark Test for SuperBench Benchmarks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants