Implement Python global state API for xray. #2125

robertnishihara · 2018-05-23T01:13:00Z

Implement most of the global state API for xray.

Note that the implementation will change a bit once we have GCS sharding.

AmplabJenkins · 2018-05-23T01:43:40Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5587/
Test FAILed.

ericl · 2018-05-23T23:07:14Z

All clients are raylets right?

AmplabJenkins · 2018-05-26T05:54:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5633/
Test PASSed.

AmplabJenkins · 2018-05-26T07:25:07Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5636/
Test PASSed.

AmplabJenkins · 2018-05-27T00:22:08Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5646/
Test FAILed.

AmplabJenkins · 2018-05-28T05:57:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5657/
Test FAILed.

AmplabJenkins · 2018-05-28T06:18:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5655/
Test PASSed.

AmplabJenkins · 2018-05-28T06:23:19Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5656/
Test PASSed.

AmplabJenkins · 2018-05-28T22:22:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5666/
Test PASSed.

AmplabJenkins · 2018-05-29T00:17:27Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5668/
Test PASSed.

AmplabJenkins · 2018-05-29T06:02:52Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5678/
Test FAILed.

AmplabJenkins · 2018-05-29T06:17:22Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5680/
Test FAILed.

robertnishihara · 2018-05-29T06:22:54Z

jenkins, retest this please

pcmoritz · 2018-05-29T06:33:28Z

test/runtest.py

    def tearDown(self):
        ray.worker.cleanup()

-    @unittest.skipIf(


pcmoritz · 2018-05-29T06:37:08Z

src/common/lib/python/common_extension.cc

+  const std::vector<ObjectID> execution_dependencies(
+      *self->execution_dependencies);
+  auto const execution_spec = ray::raylet::TaskExecutionSpecification(
+      std::move(execution_dependencies));


I think this would be equivalent to saying

auto const execution_spec = ray::raylet::TaskExecutionSpecification( *self->execution_dependencies);

without creating a temporary execution_dependencies, why not that way?

Hmm, thinking about it more, that might give a compile error and TaskExecutionSpecification would need a non-move version for this to work.

AmplabJenkins · 2018-05-29T06:52:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5681/
Test FAILed.

pcmoritz · 2018-05-29T06:57:55Z

jenkins retest this please

AmplabJenkins · 2018-05-29T08:05:33Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5685/
Test PASSed.

* master: [autoscaler] GCP node provider (ray-project#2061) [xray] Evict tasks from the lineage cache (ray-project#2152) [ASV] Add ray.init and simple Ray benchmarks (ray-project#2166) Re-encrypt key for uploading to S3 from travis to use travis-ci.com. (ray-project#2169) [rllib] Fix A3C PyTorch implementation (ray-project#2036) [JavaWorker] Do not kill local-scheduler-forked workers in RunManager.cleanup (ray-project#2151) Update Travis CI badge from travis-ci.org to travis-ci.com. (ray-project#2155) Implement Python global state API for xray. (ray-project#2125) [xray] Improve flush algorithm for the lineage cache (ray-project#2130) Fix support for actor classmethods (ray-project#2146) Add empty df test (ray-project#1879) [JavaWorker] Enable java worker support (ray-project#2094) [DataFrame] Fixing the code formatting of the tests (ray-project#2123) Update resource documentation (remove outdated limitations). (ray-project#2022) bugfix: use array redis_primary_addr out of its scope (ray-project#2139) Fix infinite retry in Push function. (ray-project#2133) [JavaWorker] Changes to the directory under src for support java worker (ray-project#2093) Integrate credis with Ray & route task table entries into credis. (ray-project#1841)

Implement global state API for xray.

99a4e50

robertnishihara mentioned this pull request May 26, 2018

[WIP] Implement task table and object table Python API for xray. #1948

Closed

Fix object table.

8b8097b

ericl mentioned this pull request May 26, 2018

[xray] [autoscaler] Fix autoscaler / raylet integration #2143

Merged

robertnishihara added 2 commits May 26, 2018 16:52

Fixes for log structure.

ac59d20

Implement cluster_resources.

27a8530

robertnishihara added 4 commits May 27, 2018 21:27

Add driver task to task table.

769e62c

Remove python flatbuffers code

52ac508

Get some global state API tests running.

ba67e94

Python linting.

f23e6a5

robertnishihara changed the title ~~[WIP] Implement Python global state API for xray.~~ Implement Python global state API for xray. May 28, 2018

robertnishihara added 2 commits May 27, 2018 22:24

Fix linting.

a365def

Fix mock modules for doc

74cd1b4

Copy over flatbuffer bindings.

36a1c91

Fix for tests.

e1bb8a9

robertnishihara added 2 commits May 28, 2018 22:27

Linting

47a6254

Fix monitor crash.

407417a

pcmoritz reviewed May 29, 2018

View reviewed changes

test/runtest.py

def tearDown(self):

ray.worker.cleanup()

@unittest.skipIf(

Copy link

Contributor

pcmoritz May 29, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yay!

pcmoritz reviewed May 29, 2018

View reviewed changes

pcmoritz approved these changes May 29, 2018

View reviewed changes

pcmoritz merged commit 6172f94 into ray-project:master May 29, 2018

pcmoritz deleted the xrayclienttable branch May 29, 2018 23:25

Implement Python global state API for xray. #2125

Implement Python global state API for xray. #2125

Uh oh!

Conversation

robertnishihara commented May 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AmplabJenkins commented May 23, 2018

Uh oh!

ericl commented May 23, 2018

Uh oh!

AmplabJenkins commented May 26, 2018

Uh oh!

AmplabJenkins commented May 26, 2018

Uh oh!

AmplabJenkins commented May 27, 2018

Uh oh!

AmplabJenkins commented May 28, 2018

Uh oh!

AmplabJenkins commented May 28, 2018

Uh oh!

AmplabJenkins commented May 28, 2018

Uh oh!

AmplabJenkins commented May 28, 2018

Uh oh!

AmplabJenkins commented May 29, 2018

Uh oh!

AmplabJenkins commented May 29, 2018

Uh oh!

AmplabJenkins commented May 29, 2018

Uh oh!

robertnishihara commented May 29, 2018

Uh oh!

pcmoritz May 29, 2018

Choose a reason for hiding this comment

Uh oh!

pcmoritz May 29, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pcmoritz May 29, 2018

Choose a reason for hiding this comment

Uh oh!

AmplabJenkins commented May 29, 2018

Uh oh!

pcmoritz commented May 29, 2018

Uh oh!

AmplabJenkins commented May 29, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

robertnishihara commented May 23, 2018 •

edited

Loading

pcmoritz May 29, 2018 •

edited

Loading