[air] add tuner user guide #26837

xwjiang2010 · 2022-07-21T17:45:21Z

Why are these changes needed?

User guide for Tuner on the Ray AIR docs.

Related issue number

Checks

I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

doc/source/ray-air/tuner.rst

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

richardliaw · 2022-07-23T00:09:52Z

doc/source/ray-air/tuner.rst

+
+Hyperparameter Tuning
+=====================
+Tuner API is the recommended way of launching hyperparameter tuning jobs with Ray Tune.


Add "what is a tuner and what does it do"?

doc/source/ray-air/user-guides.rst

doc/source/_toc.yml

# Conflicts: # doc/source/_toc.yml

Signed-off-by: Kai Fricke <kai@anyscale.com>

krfricke

LGTM let's follow up with the restore docs soon.

Also cc @richardliaw for review

richardliaw

looking good; just a couple comments.

doc/source/ray-air/doc_code/tuner.py

richardliaw · 2022-08-01T15:54:36Z

doc/source/ray-air/doc_code/tuner.py

+
+# Create Tuner
+tuner = tune.Tuner(
+    my_trainer,


how about lets use a real trainer?

It's covered in the following torch/xgboost sessions.

doc/source/ray-air/tuner.rst

richardliaw · 2022-08-01T16:12:08Z

doc/source/ray-air/tuner.rst

+    ``num_samples=2`` here will be applied to the whole suite of grid search. In other words,
+    we will generate 4 trials.
+
+For a more end-to-end example, checkout our :doc:`/ray-air/examples/analyze_tuning_results` guide.


This guide only for analysis right?

We probably want to link out to Ray Tune user guide instead?

No, it is actually an end to end example of Tuner. The title is a bit misleading.

doc/source/ray-air/tuner.rst

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

dmatrix · 2022-08-02T23:01:04Z

doc/source/ray-air/tuner.rst

+There are a couple gotchas about parameter specification when using Tuners with Trainers:
+
+- By default, configuration dictionaries and config objects will be deep-merged
+- Parameters that are duplicated in the Trainer and Tuner will be overwritten by the Tuner ``param_space``


param-space.

dmatrix · 2022-08-02T23:01:58Z

doc/source/ray-air/tuner.rst

+- **Exception:** all arguments of the :class:`RunConfig <ray.air.config.RunConfig>` and :class:`TuneConfig <ray.tune.tune_config.TuneConfig>` are inherently un-tunable.
+
+
+How to configure a Tuner


How to configure a Tuner?

dmatrix · 2022-08-02T23:03:47Z

doc/source/ray-air/tuner.rst

+- custom callbacks
+- integration with cloud storage
+
+The following we showcase some common configuration of :class:`RunConfig <ray.air.config.RunConfig>`.


In the following section, we cover and showcase some common ...

dmatrix · 2022-08-02T23:04:54Z

doc/source/ray-air/tuner.rst

+    :end-before: __basic_end__
+
+
+How do I configure a search space?


How to configure a search space?

I would leave out first person I here.

dmatrix · 2022-08-02T23:05:20Z

doc/source/ray-air/tuner.rst

+See the :class:`RunConfig API reference <ray.air.config.RunConfig>` for more details.
+
+
+How do I specify parallelism?


How to configure parallelism?

dmatrix · 2022-08-02T23:06:39Z

doc/source/ray-air/tuner.rst

+Read more about this in :ref:`tune-parallelism` section.
+
+
+How do I specify an optimization algorithm?


How to specify an optimization algorithm?

dmatrix · 2022-08-02T23:07:07Z

doc/source/ray-air/tuner.rst

+How to analyze results?
+-----------------------
+
+``Tuner.fit()`` generates a ResultGrid object. This object contains metrics, results, and checkpoints


ResultGrid

dmatrix · 2022-08-02T23:07:46Z

doc/source/ray-air/tuner.rst

+-----------------------
+
+``Tuner.fit()`` generates a ResultGrid object. This object contains metrics, results, and checkpoints
+of each trial. Below is a simple usage example:


Below is a simple example:

dmatrix · 2022-08-02T23:26:24Z

doc/source/ray-air/tuner.rst

+Advanced Tuning
+---------------
+
+Tuners also offer the possibility to apply different data preprocessing steps, as shown in the following snippet.


should this be the "capability" instead of "possibility"?

dmatrix · 2022-08-02T23:26:54Z

doc/source/ray-air/tuner.rst

+    :start-after: __tune_preprocess_start__
+    :end-before: __tune_preprocess_end__
+
+You can also sample different train/validation datasets:


Additionally, you sample different ...

dmatrix · 2022-08-02T23:29:07Z

doc/source/ray-air/tuner.rst

+Additionally, if trials fail during a tuning run, they can be retried - either from scratch or
+from the latest available checkpoint.
+
+To restore the Tuner state, you just pass the path to the experiment directory to ``Tuner.restore()``.


To restore the Tuner state, pass the path to the experiment directory as an argument to Tuner.restore(...)

dmatrix · 2022-08-02T23:31:52Z

doc/source/ray-air/tuner.rst

+
+To restore the Tuner state, you just pass the path to the experiment directory to ``Tuner.restore()``.
+
+This path can be obtained e.g. from the output of a tuning run (it's called "Result logdir"). If you


This path is obtained from the output of a tuning run, namely "Result logdir." However, if you specify a name in the :class:RunConfig <ray.air.config.RunConfig>, then it is located under ~/ray_results/<name>.

dmatrix

I'm bit confused about the diagram. Why it has Two boxes named "Trainer." We may want to clarify that in the caption.

Signed-off-by: Richard Liaw <rliaw@berkeley.edu>

richardliaw · 2022-08-03T03:29:32Z

@dmatrix addressed comments!

Signed-off-by: Richard Liaw <rliaw@berkeley.edu>

Co-authored-by: Kai Fricke <kai@anyscale.com> Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

Co-authored-by: Kai Fricke <kai@anyscale.com> Co-authored-by: Richard Liaw <rliaw@berkeley.edu> Co-authored-by: Kai Fricke <kai@anyscale.com> Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

Co-authored-by: Kai Fricke <kai@anyscale.com> Co-authored-by: Richard Liaw <rliaw@berkeley.edu> Signed-off-by: Stefan van der Kleij <s.vanderkleij@viroteq.com>

xwjiang2010 requested review from maxpumperla, pcmoritz, richardliaw, edoakes, simon-mo and ericl as code owners July 21, 2022 17:45

[air] add tuner user guide.

5a05a24

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

xwjiang2010 force-pushed the tune-doc branch from 2511ef1 to 5a05a24 Compare July 21, 2022 17:46

xwjiang2010 added 2 commits July 21, 2022 10:54

inspect result_grid.

fe8baad

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

polish

dbbf994

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

xwjiang2010 assigned richardliaw and krfricke Jul 21, 2022

Actually add to the user-guide.

00a8a6f

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

xwjiang2010 commented Jul 21, 2022

View reviewed changes

doc/source/ray-air/tuner.rst Outdated Show resolved Hide resolved

up

ff9eb54

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

xwjiang2010 force-pushed the tune-doc branch from 4bdccdd to ff9eb54 Compare July 21, 2022 21:03

add params.

c910285

Signed-off-by: Xiaowei Jiang <xwjiang2010@gmail.com>

richardliaw reviewed Jul 23, 2022

View reviewed changes

ericl reviewed Jul 23, 2022

View reviewed changes

doc/source/ray-air/user-guides.rst Outdated Show resolved Hide resolved

doc/source/_toc.yml Outdated Show resolved Hide resolved

richardliaw added this to the AIR 2.0 docs milestone Jul 27, 2022

Kai Fricke added 4 commits July 29, 2022 12:41

Merge branch 'master' into air/doc/user-guide

6f08091

# Conflicts: # doc/source/_toc.yml

Apply suggestion

b40200e

Signed-off-by: Kai Fricke <kai@anyscale.com>

Update some text

70f5063

Signed-off-by: Kai Fricke <kai@anyscale.com>

Update examples, text

ac5782d

Signed-off-by: Kai Fricke <kai@anyscale.com>

xwjiang2010 added the v2.0.0-pick label Aug 1, 2022

krfricke approved these changes Aug 1, 2022

View reviewed changes

richardliaw requested changes Aug 1, 2022

View reviewed changes

address comments

f0bdc38

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

xwjiang2010 force-pushed the tune-doc branch from 1596749 to f0bdc38 Compare August 1, 2022 16:52

Merge branch 'master' into tune-doc

c8067c1

dmatrix reviewed Aug 2, 2022

View reviewed changes

update

b7c033c

Signed-off-by: Richard Liaw <rliaw@berkeley.edu>

richardliaw added 2 commits August 3, 2022 00:38

formatting

b0b0022

Signed-off-by: Richard Liaw <rliaw@berkeley.edu>

update

033dd58

Signed-off-by: Richard Liaw <rliaw@berkeley.edu>

krfricke self-assigned this Aug 3, 2022

dmatrix approved these changes Aug 3, 2022

View reviewed changes

richardliaw changed the title ~~[air] add tuner user guide.~~ [air] add tuner user guide Aug 3, 2022

richardliaw merged commit ff2b728 into ray-project:master Aug 3, 2022

scv119 added need-cherry-pick and removed need-cherry-pick labels Aug 3, 2022

xwjiang2010 added a commit to xwjiang2010/ray that referenced this pull request Aug 3, 2022

[air] add tuner user guide (ray-project#26837)

7defba6

Co-authored-by: Kai Fricke <kai@anyscale.com> Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

scv119 added the v2.0.0-pick-done label Aug 4, 2022

xwjiang2010 deleted the tune-doc branch July 26, 2023 19:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[air] add tuner user guide #26837

[air] add tuner user guide #26837

xwjiang2010 commented Jul 21, 2022 •

edited by richardliaw

Loading

richardliaw Jul 23, 2022

krfricke left a comment

richardliaw left a comment

richardliaw Aug 1, 2022

xwjiang2010 Aug 1, 2022

richardliaw Aug 1, 2022

richardliaw Aug 1, 2022

xwjiang2010 Aug 1, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix Aug 2, 2022

dmatrix left a comment

richardliaw commented Aug 3, 2022

		- Exception: all arguments of the :class:`RunConfig <ray.air.config.RunConfig>` and :class:`TuneConfig <ray.tune.tune_config.TuneConfig>` are inherently un-tunable.


		How to configure a Tuner

		:end-before: __basic_end__


		How do I configure a search space?

		See the :class:`RunConfig API reference <ray.air.config.RunConfig>` for more details.


		How do I specify parallelism?

		Read more about this in :ref:`tune-parallelism` section.


		How do I specify an optimization algorithm?


		To restore the Tuner state, you just pass the path to the experiment directory to ``Tuner.restore()``.

		This path can be obtained e.g. from the output of a tuning run (it's called "Result logdir"). If you

[air] add tuner user guide #26837

[air] add tuner user guide #26837

Conversation

xwjiang2010 commented Jul 21, 2022 • edited by richardliaw Loading

Why are these changes needed?

Related issue number

Checks

Choose a reason for hiding this comment

krfricke left a comment

Choose a reason for hiding this comment

richardliaw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmatrix left a comment

Choose a reason for hiding this comment

richardliaw commented Aug 3, 2022

xwjiang2010 commented Jul 21, 2022 •

edited by richardliaw

Loading