Skip to content

[Autotuner] Feature: add --cpu_budget and --timeout_per_trial #2395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
luarss wants to merge 35 commits into
base: master
Choose a base branch
from
Draft

[Autotuner] Feature: add --cpu_budget and --timeout_per_trial #2395

luarss wants to merge 35 commits into from

Conversation

@luarss
Copy link
Contributor

@luarss luarss commented Sep 29, 2024 •
edited
Loading

Rationale

  • More intuitive for end users who might just want to specify the total number of cpus to be blocked.
  • timeout_per_trial is different from overall timeout.

TODO

  • Tests for --cpu_budget -> verify the timeout is hit.
  • Tests for timeout per trial and timeout -> does it actually stop the entire training?

@vvbandeira vvbandeira marked this pull request as draft October 7, 2024 19:57
@luarss luarss self-assigned this Oct 8, 2024
@luarss luarss added the autotuner Flow autotuner label Oct 8, 2024
@luarss luarss changed the title WIP: [Autotuner] Feature: add --cpu_budget and --timeout_per_trial [Autotuner] Feature: add --cpu_budget and --timeout_per_trial Oct 10, 2024
@luarss luarss marked this pull request as ready for review October 12, 2024 13:53
@luarss luarss requested a review from vvbandeira October 12, 2024 13:54
@luarss luarss marked this pull request as draft October 16, 2024 09:30
@luarss luarss marked this pull request as ready for review October 17, 2024 00:35
@oharboe
Copy link
Collaborator

oharboe commented Nov 9, 2024

The various ORFS stages have vastly different memory and CPU needs.

How does the user characterize and balance this?

@vvbandeira
Copy link
Member

vvbandeira commented Nov 11, 2024

The various ORFS stages have vastly different memory and CPU needs.

How does the user characterize and balance this?

The intended usage for these knobs is to limit experiment runtime (and consequently $$ budget). These knobs do not limit how much resources a given ORFS run has access to.

CPU budget is intended to stop the experiment after the budget is spent.
Trial timeout will limit each ORFS complete flow runtime; this way, if your experiment requires 1k ORFS runs, you know you will, at most, use 1k * timout_per_trial units of time.

@luarss luarss force-pushed the feat/autotuner-budget branch from 6c1fb2d to c15d6ed Compare December 24, 2024 03:46
@luarss luarss force-pushed the feat/autotuner-budget branch from c15d6ed to bff5e2e Compare January 10, 2025 14:20
@luarss luarss marked this pull request as draft January 10, 2025 17:36
vvbandeira
vvbandeira requested changes Jan 15, 2025
View reviewed changes
Copy link
Member

@vvbandeira vvbandeira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just an early feedback.

@luarss luarss force-pushed the feat/autotuner-budget branch 3 times, most recently from 0f3104b to 8b599a6 Compare January 21, 2025 02:24
@luarss
Copy link
Contributor Author

luarss commented Jan 22, 2025

@vvbandeira this PR is almost ready - just requires the fix from #2394 to be applied

@luarss luarss force-pushed the feat/autotuner-budget branch 2 times, most recently from f1df415 to c24ba64 Compare February 8, 2025 17:32
@luarss luarss marked this pull request as ready for review February 9, 2025 10:53
@luarss luarss requested a review from vvbandeira February 9, 2025 10:53
@luarss
add initial scaffold
cb3caa2 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix args.job syntax error
3cfffe3 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
add smoke test timeout
1be04bf 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
consistent file names and fix smoke_test_sweep
2a3ff66 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
move resources_per_trial to sweep_parser
7486249 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix timeout test and add cpu_budget logic
5f46d5d 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
add --yes to avoid prompt.
a86b771 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
add prototype for CPUBudgetSmokeTest
e5c2e3e 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
luarss added 27 commits February 11, 2025 20:43
@luarss
fix timeout
236b1b7 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
reintroduce args.jobs, and recalculate timeouts accordingly
b319e5e 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
test_trial_timeout -> test_timeout
a2dac19 
Signed-off-by: luarss <jluar@precisioninno.com>
@luarss
fix cpu_budget default to be None and reduce timeout fail
3b358fb 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
use cpu_budget default -1 to mean unlimited
bc886f4 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix smoke timeout
6e96a64 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix test timeout by timeout check in subprocess.run, also fix typo in…
c311101 
… cpubudget prompt from hrs->seconds

Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
add tooltip, update readmes
582fffe 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix black
15e2a07 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
update test suite
6e12df6 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
add --yes flag
6522488 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
do not exit code
f2b7883 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fixes to smoke_test_timeout
d05ea70 
- subprocess bug: check must be False to capture nonzero retcode.
- METRIC key not inside best_result if timeout prematurely called before trial completes (true for smoke test)

Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix test_tune_resume
27b0a73 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
shorten timeout
0468020 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix subprocess invocation
220e641 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
more generous timeout for resume check
cd4455a 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
uniquify experiment dir
d5019d1 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
Fix experiment name checks
5e27fca 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
cpubudget test fixes
26ecb7c 
- make a more reasonable # of samples to avoid jenkins worker overloading.
- fix expected_timeout

Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix lint
4c61459 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix global args, restore ihp smoke test tune
81ad4e5 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
remove cpubudget from sweep test
6361372 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
standardise cpubudget test
a06b8ae 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
standardise ref_file resume tests
cb0f283 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
standardise and fix smoke_test_timeout
e841bbc 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss
fix typo
0693b7c 
Signed-off-by: Jack Luar <jluar@precisioninno.com>
@luarss luarss force-pushed the feat/autotuner-budget branch from 0205ba7 to 0693b7c Compare February 11, 2025 20:44
@vvbandeira vvbandeira marked this pull request as draft August 22, 2025 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vvbandeira vvbandeira Awaiting requested review from vvbandeira

Requested changes must be addressed to merge this pull request.

Assignees

@luarss luarss

Labels

autotuner Flow autotuner

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@luarss @oharboe @vvbandeira