Add runtime_context to get some runtime fields in worker #4065

jovany-wang · 2019-02-15T15:54:27Z

What do these changes do?

Usage:

# in worker or driver
id = ray.get_runtime_context().current_driver_id

We add a runtime_cotext to keep some runtime fields.
In this PR, we add an attribution current_driver_id which is used in many situations as we needed.

In the future, we can add other attrs to this like current_actor_id, was_current_actor_reconstructed ant etc.

Related issue number

N/A

AmplabJenkins · 2019-02-15T18:51:20Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11997/
Test PASSed.

robertnishihara

What is the use case that you have in mind for this? Are you planning on adding more context beyond the driver ID?

There are some alternatives, e.g., discussed in #3805 of simply exposing the current task, and then the driver ID could be accessed through the task.

test/runtest.py

robertnishihara · 2019-02-15T21:20:13Z

python/ray/runtime_context.py

+
+class RuntimeContext(object):
+    """A class used for getting runtime context.
+    """


If the docstring is a single line, the """ should be on the previous line.

robertnishihara · 2019-02-15T21:22:18Z

python/ray/runtime_context.py

+
+    @property
+    def current_driver_id(self):
+        """Get current driver id for this worker or driver.


Change the wording, and formatting to

"""Get current driver ID for this worker or driver. Returns: If called by a driver, this returns the driver ID. If called in a task, return the driver ID of the associated driver. """

raulchen · 2019-02-16T03:36:29Z

@robertnishihara The purpose of this PR is to expose standard APIs for some useful runtime info, e.g., current task, driver id, redis address, etc.

Previously, users need to get these data from worker's internal fields or methods, which is likely to break as the code changes. So I'd like to propose exposing some standard APIs for them. E.g.,

ray.runtime_context.current_task()
ray.runtime_context.current_driver_id()
ray.runtime_context.redis_address()

What do you think?

AmplabJenkins · 2019-02-16T08:53:36Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12024/
Test PASSed.

AmplabJenkins · 2019-02-16T16:51:46Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12033/
Test FAILed.

AmplabJenkins · 2019-02-16T17:52:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12032/
Test PASSed.

AmplabJenkins · 2019-02-17T09:40:28Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12048/
Test FAILed.

raulchen · 2019-02-18T02:41:46Z

python/ray/__init__.py

@@ -122,6 +123,7 @@
    "remote",
    "shutdown",
    "wait",
+    "runtime_context",


Could you keep these items in alphabetical order?

Co-Authored-By: jovany-wang <kingchin1218@126.com>

jovany-wang · 2019-02-18T04:18:31Z

I have refined this PR, feel free to review it again.

AmplabJenkins · 2019-02-18T04:50:30Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12062/
Test FAILed.

AmplabJenkins · 2019-02-18T05:59:47Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12064/
Test FAILed.

raulchen · 2019-02-18T06:01:49Z

test/runtest.py

@@ -2842,3 +2842,18 @@ def f():
        return 1

    assert ray.get(f.remote()) == 1
+
+
+def test_runtime_context(shutdown_only):


we can change runtest.py/test_specific_driver_id to use runtime context, then we don't need to add this test.

AmplabJenkins · 2019-02-18T06:24:29Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12067/
Test FAILed.

AmplabJenkins · 2019-02-18T07:58:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12071/
Test FAILed.

robertnishihara

I think this is a fine approach to take. I prefer ray._get_runtime_context() since presumably this will change a bunch in the future (e.g., we will add new fields, change the name of some fields, etc).

Almost all users shouldn't need to think about the runtime context, right? Certainly the current driver ID doesn't matter for most users. So I think it makes sense for this to be somewhat less exposed.

raulchen · 2019-02-18T10:20:20Z

Sounds good. Agreed that this should be less exposed.

AmplabJenkins · 2019-02-18T12:11:39Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12080/
Test FAILed.

AmplabJenkins · 2019-02-18T13:15:49Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12083/
Test FAILed.

AmplabJenkins · 2019-02-18T21:51:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12104/
Test FAILed.

jovany-wang requested review from raulchen and robertnishihara February 15, 2019 15:55

robertnishihara reviewed Feb 15, 2019

View reviewed changes

jovany-wang force-pushed the runtime_context branch from 60f4e44 to 74c59d2 Compare February 16, 2019 15:06

raulchen reviewed Feb 18, 2019

View reviewed changes

Jovan and others added 6 commits February 18, 2019 11:07

Add runtime_context

e8fa939

Update test/runtest.py

a0549bb

Co-Authored-By: jovany-wang <kingchin1218@126.com>

Update test/runtest.py

32813cb

Co-Authored-By: jovany-wang <kingchin1218@126.com>

Update test/runtest.py

2e98d54

Co-Authored-By: jovany-wang <kingchin1218@126.com>

Address comments.

8e5b0c6

Fix lint.

cd01d46

jovany-wang force-pushed the runtime_context branch from ac964fd to cd01d46 Compare February 18, 2019 03:07

Jovan added 2 commits February 18, 2019 12:14

Refine and address comments.

c5f4efe

fix lint.

655f2c6

Address comment.

af536f5

raulchen reviewed Feb 18, 2019

View reviewed changes

Address comment.

357a7e4

robertnishihara reviewed Feb 18, 2019

View reviewed changes

Jovan added 2 commits February 18, 2019 18:27

Refine

3853ab5

fix lint

da453bf

raulchen approved these changes Feb 18, 2019

View reviewed changes

Update worker.py

cfa8ca8

raulchen merged commit 794a093 into ray-project:master Feb 19, 2019

jovany-wang deleted the runtime_context branch February 20, 2019 08:54

jovany-wang mentioned this pull request Feb 28, 2019

[Java] Add runtime context #4194

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add runtime_context to get some runtime fields in worker #4065

Add runtime_context to get some runtime fields in worker #4065

jovany-wang commented Feb 15, 2019 •

edited

Loading

AmplabJenkins commented Feb 15, 2019

robertnishihara left a comment

robertnishihara Feb 15, 2019

robertnishihara Feb 15, 2019

raulchen commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 17, 2019

raulchen Feb 18, 2019

jovany-wang commented Feb 18, 2019 •

edited

Loading

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

raulchen Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

robertnishihara left a comment

raulchen commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

Add runtime_context to get some runtime fields in worker #4065

Add runtime_context to get some runtime fields in worker #4065

Conversation

jovany-wang commented Feb 15, 2019 • edited Loading

What do these changes do?

Related issue number

AmplabJenkins commented Feb 15, 2019

robertnishihara left a comment

Choose a reason for hiding this comment

robertnishihara Feb 15, 2019

Choose a reason for hiding this comment

robertnishihara Feb 15, 2019

Choose a reason for hiding this comment

raulchen commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 17, 2019

raulchen Feb 18, 2019

Choose a reason for hiding this comment

jovany-wang commented Feb 18, 2019 • edited Loading

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

raulchen Feb 18, 2019

Choose a reason for hiding this comment

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

robertnishihara left a comment

Choose a reason for hiding this comment

raulchen commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

AmplabJenkins commented Feb 18, 2019

jovany-wang commented Feb 15, 2019 •

edited

Loading

jovany-wang commented Feb 18, 2019 •

edited

Loading