[wip][docs] update FrameworkPredictor examples #38634

matthewdeng · 2023-08-19T02:15:28Z

Why are these changes needed?

xgboost_example.ipynb and lightgbm_example.ipynb - update examples to use new flow.
1. Added Preprocessor.pickle() and Preprocessor.unpickle() utility methods to (de)serialize Preprocessors as metadata. This is copied from [train] Add documentation for using metadata argument to save preprocessors #38701
2. This depends on [train] Storage: Update XGBoost + LightGBM callback APIs #38558 in order to run with the new checkpoints.

Related issue number

Closes #38293
Depends on #38558

Checks

https://buildkite.com/ray-project/oss-ci-build-pr/builds/33973#018a38cd-40e8-4a63-9de3-ece2eb77b1f6

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Matthew Deng <matt@anyscale.com>

…ictors

Signed-off-by: Matthew Deng <matt@anyscale.com>

doc/source/ray-air/examples/lightgbm_example.ipynb

doc/source/ray-air/examples/xgboost_example.ipynb

Signed-off-by: Matthew Deng <matt@anyscale.com>

Deprecates passing in preprocessor to Trainers. Also deprecates Chain and BatchMapper preprocessors. Removes all usage of preprocessors on non-tabular data from the docs and examples. Closes #38290 Depends on #38634 and #38600 --------- Signed-off-by: amogkam <amogkamsetty@yahoo.com> Signed-off-by: Amog Kamsetty <amogkam@users.noreply.github.com> Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>

…ictors

Signed-off-by: Matthew Deng <matt@anyscale.com>

…ictors

Signed-off-by: matthewdeng <matt@anyscale.com>

Signed-off-by: Matthew Deng <matt@anyscale.com>

…ictors

Signed-off-by: Matthew Deng <matt@anyscale.com>

doc/source/ray-air/examples/lightgbm_example.ipynb

Signed-off-by: Matthew Deng <matt@anyscale.com>

Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com>

@justinvyu

* [train] enable new persistence mode for core and serve tests (#38938) Signed-off-by: Matthew Deng <matt@anyscale.com> * [train] New persistence mode: Update 🐠 `ML Libraries w/ Ray Client Examples (Python 3.7)` (#38923) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] remove non-URI assertion (#38944) Signed-off-by: Matthew Deng <matt@anyscale.com> * [train] New persistence mode: Update 📖 `Doc tests and examples (excluding Ray AIR examples)` (#38940) Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: Matthew Deng <matt@anyscale.com> Co-authored-by: Matthew Deng <matt@anyscale.com> * disable legacy sync config logic in trainable (#38952) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [2.7 CI][New Persistent Mode][6/n] 📖 ✈️ Ray AIR examples (#38918) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * [2.7 CI][New Persistent Mode][2/n] 📺 📖 Doc GPU tests and examples (#38905) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * [2.7 CI][New Persistent Mode][4/n] 📺 🚂 Train GPU tests & 🚂 Datasets Train Integration GPU Tests and Examples (#38910) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> Signed-off-by: Justin Yu <justinvyu@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> * [2.7 CI][New Persistent Mode][1/n] 📺 ✈️ AIR GPU tests (ray/air) & ⚡ :python: Lightning 2.0 Train GPU tests (#38903) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com> * [train] Fix broken tune tests and support ray storage (#38950) This PR re-introduces support for ray storage ray.init(storage="s3://...") and fixes a broken tune controller test. Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] New persistence mode: Finish migrating `xgb`, `lgbm` and `sklearn` trainers, checkpoints + tests (#38959) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [2.7 CI][New Persistent Mode][5/n] 📖 Doc examples for external code (#38915) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * [train][rllib] temporarily disable new persistence mode for rllib tests (#38965) Signed-off-by: Matthew Deng <matt@anyscale.com> * [2.7 CI][New Persistent Mode][8/n] ✈️ AIR tests (ray/air) (#38932) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * [tune] Storage: 🐙 🧠 Tune tests and examples {using RLlib} migration (#38895) Signed-off-by: Kai Fricke <kai@anyscale.com> Co-authored-by: matthewdeng <matt@anyscale.com> * [train] Fix MosaicTrainer example and unit test (#38970) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [air/release] Fix dreambooth example image preprocessing logic (#39020) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] clean up ray.train._checkpoint imports (#38951) Signed-off-by: Matthew Deng <matt@anyscale.com> * [train] high level cleanup of Ray Train docs (#38971) Signed-off-by: Matthew Deng <matt@anyscale.com> * [wip][docs] update FrameworkPredictor examples (#38634) Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com> * [train] Add documentation for using metadata argument to save preprocessors (#38701) * [Train] Restructure Ray Train Example Page (#38814) Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> * [air] Deprecate some fields/classes that are supposed to be gone in 2.6. (#38794) Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com> * [tune/storage] Fix Tune multinode tests (#39050) Fixes multinode tests by using the new train.report() API. Signed-off-by: Kai Fricke <kai@anyscale.com> * [tune] Fix BOHB example for new storage (#38983) The new storage path does not create "empty" checkpoints per default anymore. Previously, when no checkpoint is saved, PAUSEing a trial would create a dummy checkpoint that only contains trial metadata (such as the iteration number). This is not the case anymore. Examples now have to implement checkpointing to properly restore previous state. This was also true previously - but some of our simple examples (e.g. the one in this PR) didn't implement it and still "worked". I think it's fine to keep the functionality as is and require our examples to show checkpointing implementations. This will ensure that users don't shoot their feet trying to use e.g. BOHB. Separately, BOHB was malfunctioning as trials were repeatedly PAUSED and restarted as they've never been removed from `bracket.trials_to_unpause`. @justinvyu mentioned this in the review where it was introduced and I believed at the time it wasn't necessary - turns out it is, as we can end up in a situation where a bracket is never finished because trials are constantly running. This was not caught by any tests. We should add one in a follow-up - for now we can proceed with this PR to pick onto Ray 2.7. Signed-off-by: Kai Fricke <kai@anyscale.com> * [Release Test] Fix `long_running_horovod_tune_test`. (#39012) Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com> Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com> * [train] New persistence mode: `StorageContext` unit tests (#39023) Signed-off-by: Justin Yu <justinvyu@anyscale.com> * [train] enable train + tune tests and examples (#39021) Signed-off-by: Matthew Deng <matt@anyscale.com> * [rllib] Fix storage-path related tests (#38947) This PR fixes rllib-related tests that didn't pass changes related to the new storage context. Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com> Co-authored-by: matthewdeng <matt@anyscale.com> * [train] New persistence mode: Migrate 🐙 `Tune tests and examples (medium)` (#39081) Signed-off-by: Justin Yu <justinvyu@anyscale.com> --------- Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: Justin Yu <justinvyu@anyscale.com> Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com> Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com> Signed-off-by: Kai Fricke <kai@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com> Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com> Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com> Co-authored-by: Justin Yu <justinvyu@anyscale.com> Co-authored-by: Yunxuan Xiao <yunxuanx@anyscale.com> Co-authored-by: Kai Fricke <krfricke@users.noreply.github.com> Co-authored-by: Eric Liang <ekhliang@gmail.com> Co-authored-by: xwjiang2010 <87673679+xwjiang2010@users.noreply.github.com>

Deprecates passing in preprocessor to Trainers. Also deprecates Chain and BatchMapper preprocessors. Removes all usage of preprocessors on non-tabular data from the docs and examples. Closes ray-project#38290 Depends on ray-project#38634 and ray-project#38600 --------- Signed-off-by: amogkam <amogkamsetty@yahoo.com> Signed-off-by: Amog Kamsetty <amogkam@users.noreply.github.com> Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>

Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com>

Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com> Signed-off-by: Jim Thompson <jimthompson5802@gmail.com>

Deprecates passing in preprocessor to Trainers. Also deprecates Chain and BatchMapper preprocessors. Removes all usage of preprocessors on non-tabular data from the docs and examples. Closes ray-project#38290 Depends on ray-project#38634 and ray-project#38600 --------- Signed-off-by: amogkam <amogkamsetty@yahoo.com> Signed-off-by: Amog Kamsetty <amogkam@users.noreply.github.com> Co-authored-by: matthewdeng <matthew.j.deng@gmail.com> Signed-off-by: Victor <vctr.y.m@example.com>

Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>

matthewdeng added 3 commits August 18, 2023 14:28

wip

b9dea85

Signed-off-by: Matthew Deng <matt@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into cleanup-pred…

e8d5e31

…ictors

[docs] update FrameworkPredictors

35c25ea

Signed-off-by: Matthew Deng <matt@anyscale.com>

amogkam mentioned this pull request Aug 21, 2023

[Train] Deprecate preprocessor arg to Trainer #38640

Merged

8 tasks

amogkam reviewed Aug 21, 2023

View reviewed changes

doc/source/ray-air/examples/lightgbm_example.ipynb Outdated Show resolved Hide resolved

doc/source/ray-air/examples/xgboost_example.ipynb Outdated Show resolved Hide resolved

matthewdeng added 2 commits August 21, 2023 21:05

move preprocessing out

81720f3

Signed-off-by: Matthew Deng <matt@anyscale.com>

np

95c34df

Signed-off-by: Matthew Deng <matt@anyscale.com>

matthewdeng added 10 commits August 24, 2023 11:52

Merge branch 'master' of github.com:ray-project/ray into cleanup-pred…

16b78be

…ictors

update

7fa74a8

Signed-off-by: Matthew Deng <matt@anyscale.com>

consistnecy

0b0cbd7

Signed-off-by: Matthew Deng <matt@anyscale.com>

fix

dc3cc39

Signed-off-by: Matthew Deng <matt@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into cleanup-pred…

62c0560

…ictors

Merge branch 'master' into cleanup-predictors

26396d7

Signed-off-by: matthewdeng <matt@anyscale.com>

enable

e0140b2

Signed-off-by: Matthew Deng <matt@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into cleanup-pred…

d56fea2

…ictors

fix

3a4c538

Signed-off-by: Matthew Deng <matt@anyscale.com>

import

94d4e8b

Signed-off-by: Matthew Deng <matt@anyscale.com>

matthewdeng marked this pull request as ready for review August 27, 2023 23:52

matthewdeng requested review from richardliaw, gjoliver, krfricke, xwjiang2010, Yard1, maxpumperla, a team, ericl, scv119, c21 and scottjlee as code owners August 27, 2023 23:52

matthewdeng requested review from bveeramani and raulchen as code owners August 27, 2023 23:52

matthewdeng assigned ericl and amogkam Aug 27, 2023

matthewdeng added tests-ok The tagger certifies test failures are unrelated and assumes personal liability. v2.7.0-pick labels Aug 28, 2023

ericl reviewed Aug 28, 2023

View reviewed changes

doc/source/ray-air/examples/lightgbm_example.ipynb Show resolved Hide resolved

ericl reviewed Aug 28, 2023

View reviewed changes

doc/source/ray-air/examples/lightgbm_example.ipynb Show resolved Hide resolved

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 28, 2023

undeprecate chain

566435c

Signed-off-by: Matthew Deng <matt@anyscale.com>

matthewdeng removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 28, 2023

ericl approved these changes Aug 28, 2023

View reviewed changes

matthewdeng merged commit 59515e1 into ray-project:master Aug 29, 2023

matthewdeng added a commit to matthewdeng/ray that referenced this pull request Aug 30, 2023

[wip][docs] update FrameworkPredictor examples (ray-project#38634)

e0225a7

Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com>

LeonLuttenberger pushed a commit to jaidisido/ray that referenced this pull request Sep 5, 2023

[wip][docs] update FrameworkPredictor examples (ray-project#38634)

2850b9d

Signed-off-by: Matthew Deng <matt@anyscale.com> Signed-off-by: matthewdeng <matt@anyscale.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[wip][docs] update FrameworkPredictor examples #38634

[wip][docs] update FrameworkPredictor examples #38634

Uh oh!

matthewdeng commented Aug 19, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

[wip][docs] update FrameworkPredictor examples #38634

[wip][docs] update FrameworkPredictor examples #38634

Uh oh!

Conversation

matthewdeng commented Aug 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matthewdeng commented Aug 19, 2023 •

edited

Loading