Dynamic Custom Resources - create and delete resources #3742

romilbhardwaj · 2019-01-10T15:49:41Z

What do these changes do?

This PR implements support for dynamic custom resources, allowing creation and removal of custom resources at runtime. This is the expected API:

ray.experimental.set_resource(resource_name, capacity, clientId=None)
Sets a resource for a target clientId. If clientId is not specified, sets the resource on the local node. If the resource already exists, the capacity is updated. If capacity is set to 0, the resource is deleted.

Related issue number

#3432

AmplabJenkins · 2019-01-10T18:27:49Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10748/
Test PASSed.

robertnishihara · 2019-01-12T06:17:04Z

@romilbhardwaj Can you add some tests as well? E.g., in runtest.py.

raulchen · 2019-01-13T14:46:59Z

Regarding the API, I think maybe we can combine the 2 APIs into one, say update_resource. If users pass in 0 as the capacity, then we delete the resource. Also, because this API isn't supposed to be used to normal users, maybe it's better to put it under the ray.internal module.

raulchen · 2019-01-13T14:54:55Z

Also, if I remember correctly, current client table only has 1 NIL-id key. And each client has 1 or 2 log entries in the z-set. This is fine for now, because the number of clients isn't too large usually.
However, if we append every amend operation as an entry into the log, the z-set could become huge. Thus, I think it's better to make the table one key per client id. And we can add another ClientResourceTable for that.

AmplabJenkins · 2019-01-15T00:42:11Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10862/
Test FAILed.

romilbhardwaj · 2019-01-15T01:02:26Z

Thanks for the inputs @raulchen. Some thoughts:

I think setting a resource to zero is not equivalent to deleting it. A zero capacity resource could be used as a label and tasks requiring 0 units of that resource will get scheduled on the node with the resource.
I agree, we should move this out of ray.worker. Since this is a new API, I think it's good to keep it in ray.experimental for now.
One key per client id is a good idea, considering frequent resource addition/deletions will rapidly increase the log size. We should eventually do this, but I think this would require changes across the code base wherever the client table is accessed or modified. I feel this is best done in a separate PR.

AmplabJenkins · 2019-01-15T01:04:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10865/
Test FAILed.

AmplabJenkins · 2019-01-15T02:43:35Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/10861/
Test PASSed.

raulchen · 2019-01-16T07:54:59Z

I think setting a resource to zero is not equivalent to deleting it. A zero capacity resource could be used as a label and tasks requiring 0 units of that resource will get scheduled on the node with the resource.

For simplicity, I actually prefer treating "A node having 0 unit of a resource" and "A node not having a resource" as the same thing. Otherwise, it might be a bit confusing. Is there any reason why distinguishing these 2 cases is better?

I agree, we should move this out of ray.worker. Since this is a new API, I think it's good to keep it in ray.experimental for now.

ray.experimental is fine as well.

One key per client id is a good idea, considering frequent resource addition/deletions will rapidly increase the log size. We should eventually do this, but I think this would require changes across the code base wherever the client table is accessed or modified. I feel this is best done in a separate PR.

That doesn't seem to require too many changes in existing code base. It looks to me that we only need to:

publish resource data to a ClientResourceTable, instead of ClientTable.
change the NodeAdded/NodeRemoved callbacks in node manager.
change the cluster_resources in state.py.

Did I miss something?

Another thing I forgot to mention is that the updating resources is now async, but users may want a sync API. For example, if I remove a resource from a node and then submit a task with that resource, I definitely expect the task not to be scheduled to the node.

jovany-wang · 2019-01-25T07:26:10Z

Hi, @romilbhardwaj
Any progress of this PR?

robertnishihara · 2019-02-05T19:01:11Z

I think setting a resource to zero is not equivalent to deleting it. A zero capacity resource could be used as a label and tasks requiring 0 units of that resource will get scheduled on the node with the resource.

For simplicity, I actually prefer treating "A node having 0 unit of a resource" and "A node not having a resource" as the same thing. Otherwise, it might be a bit confusing. Is there any reason why distinguishing these 2 cases is better?

@romilbhardwaj @raulchen Yes, let's please make a "zero capacity resource" the exact same thing as "a node not having a resource".

AmplabJenkins · 2019-02-08T23:12:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/11705/
Test FAILed.

virtualluke · 2019-02-13T03:32:54Z

Does this also change the cap of a resource to be higher than 512? Is there a status on this PR?

romilbhardwaj · 2019-02-14T01:39:17Z

@virtualluke Yes, this should allow you to create more than 512 resources. However, we are yet to test the effects of having large number of resources on performance. I'm testing the implementation now and ironing out some bugs - will submit it for review soon.

AmplabJenkins · 2019-02-16T03:24:56Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12016/
Test FAILed.

AmplabJenkins · 2019-02-16T03:56:16Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/12018/
Test FAILed.

AmplabJenkins · 2019-05-01T03:02:41Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14026/
Test FAILed.

AmplabJenkins · 2019-05-01T21:23:19Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14038/
Test FAILed.

AmplabJenkins · 2019-05-01T23:06:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14041/
Test FAILed.

AmplabJenkins · 2019-05-02T00:09:59Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14042/
Test FAILed.

raulchen

Thanks for working on this. Looks to me overall, except that the perf issue I mentioned above might still be a problem. But I think that can be improved later.

@robertnishihara @guoyuhong do you also want to take a look?

src/ray/raylet/format/node_manager.fbs

src/ray/raylet/raylet_client.cc

raulchen · 2019-05-08T11:37:55Z

python/ray/tests/test_dynres.py

+        return 1
+
+    task = f._remote(
+        args=[], resources={res_name: res_capacity})  # This is infeasible


_remote is an internal api and may change later. why not just use @ray.remote(resources={})?

raulchen · 2019-05-08T11:44:33Z

python/ray/tests/test_dynres.py

+    assert successful  # The task completed
+
+
+def test_dynamic_res_updation_clientid(ray_start_cluster):


Nit, I think test_dynamic_res_updation_clientid, test_dynamic_res_creation_clientid, and test_dynamic_res_deletion_clientid can be merged into one unit test, to remove some duplicated code.

raulchen

the java CI failure is related. https://travis-ci.com/ray-project/ray/jobs/197021026

java/runtime/src/main/java/org/ray/runtime/gcs/GcsClient.java:66: error: cannot find symbol
      if (data.isInsertion()) {
              ^
  symbol:   method isInsertion()
  location: variable data of type ClientTableData

guoyuhong

Thanks for the contribution! I left a few comments.

python/ray/experimental/dynamic_resources.py

src/ray/raylet/node_manager.cc

guoyuhong · 2019-05-09T05:02:36Z

src/ray/gcs/tables.h

@@ -677,7 +677,7 @@ using ConfigTable = Table<ConfigID, ConfigTableData>;
 /// it should append an entry to the log indicating that it is dead. A client
 /// that is marked as dead should never again be marked as alive; if it needs
 /// to reconnect, it must connect with a different ClientID.
-class ClientTable : private Log<ClientID, ClientTableData> {
+class ClientTable : public Log<ClientID, ClientTableData> {


Is it necessary to change private to public?

src/ray/gcs/tables.cc

java/BUILD.bazel

AmplabJenkins · 2019-05-10T04:29:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14160/
Test FAILed.

AmplabJenkins · 2019-05-10T05:36:42Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14164/
Test FAILed.

AmplabJenkins · 2019-05-10T05:46:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/14162/
Test FAILed.

* [rllib] Remove dependency on TensorFlow (ray-project#4764) * remove hard tf dep * add test * comment fix * fix test * Dynamic Custom Resources - create and delete resources (ray-project#3742) * Update tutorial link in doc (ray-project#4777) * [rllib] Implement learn_on_batch() in torch policy graph * Fix `ray stop` by killing raylet before plasma (ray-project#4778) * Fatal check if object store dies (ray-project#4763) * [rllib] fix clip by value issue as TF upgraded (ray-project#4697) * fix clip_by_value issue * fix typo * [autoscaler] Fix submit (ray-project#4782) * Queue tasks in the raylet in between async callbacks (ray-project#4766) * Add a SWAP TaskQueue so that we can keep track of tasks that are temporarily dequeued * Fix bug where tasks that fail to be forwarded don't appear to be local by adding them to SWAP queue * cleanups * updates * updates * [Java][Bazel] Refine auto-generated pom files (ray-project#4780) * Bump version to 0.7.0 (ray-project#4791) * [JAVA] setDefaultUncaughtExceptionHandler to log uncaught exception in user thread. (ray-project#4798) * Add WorkerUncaughtExceptionHandler * Fix * revert bazel and pom * [tune] Fix CLI test (ray-project#4801) * Fix pom file generation (ray-project#4800) * [rllib] Support continuous action distributions in IMPALA/APPO (ray-project#4771) * [rllib] TensorFlow 2 compatibility (ray-project#4802) * Change tagline in documentation and README. (ray-project#4807) * Update README.rst, index.rst, tutorial.rst and _config.yml * [tune] Support non-arg submit (ray-project#4803) * [autoscaler] rsync cluster (ray-project#4785) * [tune] Remove extra parsing functionality (ray-project#4804) * Fix Java worker log dir (ray-project#4781) * [tune] Initial track integration (ray-project#4362) Introduces a minimally invasive utility for logging experiment results. A broad requirement for this tool is that it should integrate seamlessly with Tune execution. * [rllib] [RFC] Dynamic definition of loss functions and modularization support (ray-project#4795) * dynamic graph * wip * clean up * fix * document trainer * wip * initialize the graph using a fake batch * clean up dynamic init * wip * spelling * use builder for ppo pol graph * add ppo graph * fix naming * order * docs * set class name correctly * add torch builder * add custom model support in builder * cleanup * remove underscores * fix py2 compat * Update dynamic_tf_policy_graph.py * Update tracking_dict.py * wip * rename * debug level * rename policy_graph -> policy in new classes * fix test * rename ppo tf policy * port appo too * forgot grads * default policy optimizer * make default config optional * add config to optimizer * use lr by default in optimizer * update * comments * remove optimizer * fix tuple actions support in dynamic tf graph * [rllib] Rename PolicyGraph => Policy, move from evaluation/ to policy/ (ray-project#4819) This implements some of the renames proposed in ray-project#4813 We leave behind backwards-compatibility aliases for *PolicyGraph and SampleBatch. * [Java] Dynamic resource API in Java (ray-project#4824) * Add default values for Wgym flags * Fix import * Fix issue when starting `raylet_monitor` (ray-project#4829) * Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID (ray-project#4776) * Enable BaseId. * Change TaskID and make python test pass * Remove unnecessary functions and fix test failure and change TaskID to 16 bytes. * Java code change draft * Refine * Lint * Update java/api/src/main/java/org/ray/api/id/TaskId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/ObjectId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comment * Lint * Fix SINGLE_PROCESS * Fix comments * Refine code * Refine test * Resolve conflict * Fix bug in which actor classes are not exported multiple times. (ray-project#4838) * Bump Ray master version to 0.8.0.dev0 (ray-project#4845) * Add section to bump version of master branch and cleanup release docs (ray-project#4846) * Fix import * Export remote functions when first used and also fix bug in which rem… (ray-project#4844) * Export remote functions when first used and also fix bug in which remote functions and actor classes are not exported from workers during subsequent ray sessions. * Documentation update * Fix tests. * Fix grammar * Update wheel versions in documentation to 0.8.0.dev0 and 0.7.0. (ray-project#4847) * [tune] Later expansion of local_dir (ray-project#4806) * [rllib] [RFC] Deprecate Python 2 / RLlib (ray-project#4832) * Fix a typo in kubernetes yaml (ray-project#4872) * Move global state API out of global_state object. (ray-project#4857) * Install bazel in autoscaler development configs. (ray-project#4874) * [tune] Fix up Ax Search and Examples (ray-project#4851) * update Ax for cleaner API * docs update * [rllib] Update concepts docs and add "Building Policies in Torch/TensorFlow" section (ray-project#4821) * wip * fix index * fix bugs * todo * add imports * note on get ph * note on get ph * rename to building custom algs * add rnn state info * [rllib] Fix error getting kl when simple_optimizer: True in multi-agent PPO * Replace ReturnIds with NumReturns in TaskInfo to reduce the size (ray-project#4854) * Refine TaskInfo * Fix * Add a test to print task info size * Lint * Refine * Update deps commits of opencensus to support building with bzl 0.25.x (ray-project#4862) * Update deps to support bzl 2.5.x * Fix

* [rllib] Remove dependency on TensorFlow (ray-project#4764) * remove hard tf dep * add test * comment fix * fix test * Dynamic Custom Resources - create and delete resources (ray-project#3742) * Update tutorial link in doc (ray-project#4777) * [rllib] Implement learn_on_batch() in torch policy graph * Fix `ray stop` by killing raylet before plasma (ray-project#4778) * Fatal check if object store dies (ray-project#4763) * [rllib] fix clip by value issue as TF upgraded (ray-project#4697) * fix clip_by_value issue * fix typo * [autoscaler] Fix submit (ray-project#4782) * Queue tasks in the raylet in between async callbacks (ray-project#4766) * Add a SWAP TaskQueue so that we can keep track of tasks that are temporarily dequeued * Fix bug where tasks that fail to be forwarded don't appear to be local by adding them to SWAP queue * cleanups * updates * updates * [Java][Bazel] Refine auto-generated pom files (ray-project#4780) * Bump version to 0.7.0 (ray-project#4791) * [JAVA] setDefaultUncaughtExceptionHandler to log uncaught exception in user thread. (ray-project#4798) * Add WorkerUncaughtExceptionHandler * Fix * revert bazel and pom * [tune] Fix CLI test (ray-project#4801) * Fix pom file generation (ray-project#4800) * [rllib] Support continuous action distributions in IMPALA/APPO (ray-project#4771) * [rllib] TensorFlow 2 compatibility (ray-project#4802) * Change tagline in documentation and README. (ray-project#4807) * Update README.rst, index.rst, tutorial.rst and _config.yml * [tune] Support non-arg submit (ray-project#4803) * [autoscaler] rsync cluster (ray-project#4785) * [tune] Remove extra parsing functionality (ray-project#4804) * Fix Java worker log dir (ray-project#4781) * [tune] Initial track integration (ray-project#4362) Introduces a minimally invasive utility for logging experiment results. A broad requirement for this tool is that it should integrate seamlessly with Tune execution. * [rllib] [RFC] Dynamic definition of loss functions and modularization support (ray-project#4795) * dynamic graph * wip * clean up * fix * document trainer * wip * initialize the graph using a fake batch * clean up dynamic init * wip * spelling * use builder for ppo pol graph * add ppo graph * fix naming * order * docs * set class name correctly * add torch builder * add custom model support in builder * cleanup * remove underscores * fix py2 compat * Update dynamic_tf_policy_graph.py * Update tracking_dict.py * wip * rename * debug level * rename policy_graph -> policy in new classes * fix test * rename ppo tf policy * port appo too * forgot grads * default policy optimizer * make default config optional * add config to optimizer * use lr by default in optimizer * update * comments * remove optimizer * fix tuple actions support in dynamic tf graph * [rllib] Rename PolicyGraph => Policy, move from evaluation/ to policy/ (ray-project#4819) This implements some of the renames proposed in ray-project#4813 We leave behind backwards-compatibility aliases for *PolicyGraph and SampleBatch. * [Java] Dynamic resource API in Java (ray-project#4824) * Add default values for Wgym flags * Fix import * Fix issue when starting `raylet_monitor` (ray-project#4829) * Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID (ray-project#4776) * Enable BaseId. * Change TaskID and make python test pass * Remove unnecessary functions and fix test failure and change TaskID to 16 bytes. * Java code change draft * Refine * Lint * Update java/api/src/main/java/org/ray/api/id/TaskId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/ObjectId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comment * Lint * Fix SINGLE_PROCESS * Fix comments * Refine code * Refine test * Resolve conflict * Fix bug in which actor classes are not exported multiple times. (ray-project#4838) * Bump Ray master version to 0.8.0.dev0 (ray-project#4845) * Add section to bump version of master branch and cleanup release docs (ray-project#4846) * Fix import * Export remote functions when first used and also fix bug in which rem… (ray-project#4844) * Export remote functions when first used and also fix bug in which remote functions and actor classes are not exported from workers during subsequent ray sessions. * Documentation update * Fix tests. * Fix grammar * Update wheel versions in documentation to 0.8.0.dev0 and 0.7.0. (ray-project#4847) * [tune] Later expansion of local_dir (ray-project#4806) * [rllib] [RFC] Deprecate Python 2 / RLlib (ray-project#4832) * Fix a typo in kubernetes yaml (ray-project#4872) * Move global state API out of global_state object. (ray-project#4857) * Install bazel in autoscaler development configs. (ray-project#4874) * [tune] Fix up Ax Search and Examples (ray-project#4851) * update Ax for cleaner API * docs update * [rllib] Update concepts docs and add "Building Policies in Torch/TensorFlow" section (ray-project#4821) * wip * fix index * fix bugs * todo * add imports * note on get ph * note on get ph * rename to building custom algs * add rnn state info * [rllib] Fix error getting kl when simple_optimizer: True in multi-agent PPO * Replace ReturnIds with NumReturns in TaskInfo to reduce the size (ray-project#4854) * Refine TaskInfo * Fix * Add a test to print task info size * Lint * Refine * Update deps commits of opencensus to support building with bzl 0.25.x (ray-project#4862) * Update deps to support bzl 2.5.x * Fix * Upgrade arrow to latest master (ray-project#4858) * [tune] Auto-init Ray + default SearchAlg (ray-project#4815) * Bump version from 0.8.0.dev0 to 0.7.1. (ray-project#4890) * [rllib] Allow access to batches prior to postprocessing (ray-project#4871) * [rllib] Fix Multidiscrete support (ray-project#4869) * Refactor redis callback handling (ray-project#4841) * Add CallbackReply * Fix * fix linting by format.sh * Fix linting * Address comments. * Fix * Initial high-level code structure of CoreWorker. (ray-project#4875) * Drop duplicated string format (ray-project#4897) This string format is unnecessary. java_worker_options has been appended to the commandline later. * Refactor ID Serial 2: change all ID functions to `CamelCase` (ray-project#4896) * Hotfix for change of from_random to FromRandom (ray-project#4909) * [rllib] Fix documentation on custom policies (ray-project#4910) * wip * add docs * lint * todo sections * fix doc * [rllib] Allow Torch policies access to full action input dict in extra_action_out_fn (ray-project#4894) * fix torch extra out * preserve setitem * fix docs * [tune] Pretty print params json in logger.py (ray-project#4903) * [sgd] Distributed Training via PyTorch (ray-project#4797) Implements distributed SGD using distributed PyTorch. * [rllib] Rough port of DQN to build_tf_policy() pattern (ray-project#4823) * fetching objects in parallel in _get_arguments_for_execution (ray-project#4775) * [tune] Disallow setting resources_per_trial when it is already configured (ray-project#4880) * disallow it * import fix * fix example * fix test * fix tests * Update mock.py * fix * make less convoluted * fix tests * [rllib] Rename PolicyEvaluator => RolloutWorker (ray-project#4820) * Fix local cluster yaml (ray-project#4918) * [tune] Directional metrics for components (ray-project#4120) (ray-project#4915) * [Core Worker] implement ObjectInterface and add test framework (ray-project#4899) * [tune] Make PBT Quantile fraction configurable (ray-project#4912) * Better organize ray_common module (ray-project#4898) * Fix error * Fix compute actions return value

kfstorm · 2019-06-14T05:08:44Z

src/ray/raylet/node_manager.cc

+    const std::string &resource_label = resource_pair.first;
+    const double &new_resource_capacity = resource_pair.second;
+
+    cluster_schedres.UpdateResource(resource_label, new_resource_capacity);


@romilbhardwaj Is this a bug? new_resource_capacity is of type double. But the second parameter of UpdateResource is actually int64_t.

* [rllib] Remove dependency on TensorFlow (ray-project#4764) * remove hard tf dep * add test * comment fix * fix test * Dynamic Custom Resources - create and delete resources (ray-project#3742) * Update tutorial link in doc (ray-project#4777) * [rllib] Implement learn_on_batch() in torch policy graph * Fix `ray stop` by killing raylet before plasma (ray-project#4778) * Fatal check if object store dies (ray-project#4763) * [rllib] fix clip by value issue as TF upgraded (ray-project#4697) * fix clip_by_value issue * fix typo * [autoscaler] Fix submit (ray-project#4782) * Queue tasks in the raylet in between async callbacks (ray-project#4766) * Add a SWAP TaskQueue so that we can keep track of tasks that are temporarily dequeued * Fix bug where tasks that fail to be forwarded don't appear to be local by adding them to SWAP queue * cleanups * updates * updates * [Java][Bazel] Refine auto-generated pom files (ray-project#4780) * Bump version to 0.7.0 (ray-project#4791) * [JAVA] setDefaultUncaughtExceptionHandler to log uncaught exception in user thread. (ray-project#4798) * Add WorkerUncaughtExceptionHandler * Fix * revert bazel and pom * [tune] Fix CLI test (ray-project#4801) * Fix pom file generation (ray-project#4800) * [rllib] Support continuous action distributions in IMPALA/APPO (ray-project#4771) * [rllib] TensorFlow 2 compatibility (ray-project#4802) * Change tagline in documentation and README. (ray-project#4807) * Update README.rst, index.rst, tutorial.rst and _config.yml * [tune] Support non-arg submit (ray-project#4803) * [autoscaler] rsync cluster (ray-project#4785) * [tune] Remove extra parsing functionality (ray-project#4804) * Fix Java worker log dir (ray-project#4781) * [tune] Initial track integration (ray-project#4362) Introduces a minimally invasive utility for logging experiment results. A broad requirement for this tool is that it should integrate seamlessly with Tune execution. * [rllib] [RFC] Dynamic definition of loss functions and modularization support (ray-project#4795) * dynamic graph * wip * clean up * fix * document trainer * wip * initialize the graph using a fake batch * clean up dynamic init * wip * spelling * use builder for ppo pol graph * add ppo graph * fix naming * order * docs * set class name correctly * add torch builder * add custom model support in builder * cleanup * remove underscores * fix py2 compat * Update dynamic_tf_policy_graph.py * Update tracking_dict.py * wip * rename * debug level * rename policy_graph -> policy in new classes * fix test * rename ppo tf policy * port appo too * forgot grads * default policy optimizer * make default config optional * add config to optimizer * use lr by default in optimizer * update * comments * remove optimizer * fix tuple actions support in dynamic tf graph * [rllib] Rename PolicyGraph => Policy, move from evaluation/ to policy/ (ray-project#4819) This implements some of the renames proposed in ray-project#4813 We leave behind backwards-compatibility aliases for *PolicyGraph and SampleBatch. * [Java] Dynamic resource API in Java (ray-project#4824) * Add default values for Wgym flags * Fix import * Fix issue when starting `raylet_monitor` (ray-project#4829) * Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID (ray-project#4776) * Enable BaseId. * Change TaskID and make python test pass * Remove unnecessary functions and fix test failure and change TaskID to 16 bytes. * Java code change draft * Refine * Lint * Update java/api/src/main/java/org/ray/api/id/TaskId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/ObjectId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comment * Lint * Fix SINGLE_PROCESS * Fix comments * Refine code * Refine test * Resolve conflict * Fix bug in which actor classes are not exported multiple times. (ray-project#4838) * Bump Ray master version to 0.8.0.dev0 (ray-project#4845) * Add section to bump version of master branch and cleanup release docs (ray-project#4846) * Fix import * Export remote functions when first used and also fix bug in which rem… (ray-project#4844) * Export remote functions when first used and also fix bug in which remote functions and actor classes are not exported from workers during subsequent ray sessions. * Documentation update * Fix tests. * Fix grammar * Update wheel versions in documentation to 0.8.0.dev0 and 0.7.0. (ray-project#4847) * [tune] Later expansion of local_dir (ray-project#4806) * [rllib] [RFC] Deprecate Python 2 / RLlib (ray-project#4832) * Fix a typo in kubernetes yaml (ray-project#4872) * Move global state API out of global_state object. (ray-project#4857) * Install bazel in autoscaler development configs. (ray-project#4874) * [tune] Fix up Ax Search and Examples (ray-project#4851) * update Ax for cleaner API * docs update * [rllib] Update concepts docs and add "Building Policies in Torch/TensorFlow" section (ray-project#4821) * wip * fix index * fix bugs * todo * add imports * note on get ph * note on get ph * rename to building custom algs * add rnn state info * [rllib] Fix error getting kl when simple_optimizer: True in multi-agent PPO * Replace ReturnIds with NumReturns in TaskInfo to reduce the size (ray-project#4854) * Refine TaskInfo * Fix * Add a test to print task info size * Lint * Refine * Update deps commits of opencensus to support building with bzl 0.25.x (ray-project#4862) * Update deps to support bzl 2.5.x * Fix * Upgrade arrow to latest master (ray-project#4858) * [tune] Auto-init Ray + default SearchAlg (ray-project#4815) * Bump version from 0.8.0.dev0 to 0.7.1. (ray-project#4890) * [rllib] Allow access to batches prior to postprocessing (ray-project#4871) * [rllib] Fix Multidiscrete support (ray-project#4869) * Refactor redis callback handling (ray-project#4841) * Add CallbackReply * Fix * fix linting by format.sh * Fix linting * Address comments. * Fix * Initial high-level code structure of CoreWorker. (ray-project#4875) * Drop duplicated string format (ray-project#4897) This string format is unnecessary. java_worker_options has been appended to the commandline later. * Refactor ID Serial 2: change all ID functions to `CamelCase` (ray-project#4896) * Hotfix for change of from_random to FromRandom (ray-project#4909) * [rllib] Fix documentation on custom policies (ray-project#4910) * wip * add docs * lint * todo sections * fix doc * [rllib] Allow Torch policies access to full action input dict in extra_action_out_fn (ray-project#4894) * fix torch extra out * preserve setitem * fix docs * [tune] Pretty print params json in logger.py (ray-project#4903) * [sgd] Distributed Training via PyTorch (ray-project#4797) Implements distributed SGD using distributed PyTorch. * [rllib] Rough port of DQN to build_tf_policy() pattern (ray-project#4823) * fetching objects in parallel in _get_arguments_for_execution (ray-project#4775) * [tune] Disallow setting resources_per_trial when it is already configured (ray-project#4880) * disallow it * import fix * fix example * fix test * fix tests * Update mock.py * fix * make less convoluted * fix tests * [rllib] Rename PolicyEvaluator => RolloutWorker (ray-project#4820) * Fix local cluster yaml (ray-project#4918) * [tune] Directional metrics for components (ray-project#4120) (ray-project#4915) * [Core Worker] implement ObjectInterface and add test framework (ray-project#4899) * [tune] Make PBT Quantile fraction configurable (ray-project#4912) * Better organize ray_common module (ray-project#4898) * Fix error * [tune] Add requirements-dev.txt and update docs for contributing (ray-project#4925) * Add requirements-dev.txt and update docs. * Update doc/source/tune-contrib.rst Co-Authored-By: Richard Liaw <rliaw@berkeley.edu> * Unpin everything except for yapf. * Fix compute actions return value * Bump version from 0.7.1 to 0.8.0.dev1. (ray-project#4937) * Update version number in documentation after release 0.7.0 -> 0.7.1 and 0.8.0.dev0 -> 0.8.0.dev1. (ray-project#4941) * [doc] Update developer docs with bazel instructions (ray-project#4944) * [C++] Add hash table to Redis-Module (ray-project#4911) * Flush lineage cache on task submission instead of execution (ray-project#4942) * [rllib] Add docs on how to use TF eager execution (ray-project#4927) * [rllib] Port remainder of algorithms to build_trainer() pattern (ray-project#4920) * Fix resource bookkeeping bug with acquiring unknown resource. (ray-project#4945) * Update aws keys for uploading wheels to s3. (ray-project#4948) * Upload wheels on Travis to branchname/commit_id. (ray-project#4949) * [Java] Fix serializing issues of `RaySerializer` (ray-project#4887) * Fix * Address comment. * fix (ray-project#4950) * [Java] Add inner class `Builder` to build call options. (ray-project#4956) * Add Builder class * format * Refactor by IDE * Remove uncessary dependency * Make release stress tests work and improve them. (ray-project#4955) * Use proper session directory for debug_string.txt (ray-project#4960) * [core] Use int64_t instead of int to keep track of fractional resources (ray-project#4959) * [core worker] add task submission & execution interface (ray-project#4922) * [sgd] Add non-distributed PyTorch runner (ray-project#4933) * Add non-distributed PyTorch runner * use dist.is_available() instead of checking OS * Nicer exception * Fix bug in choosing port * Refactor some code * Address comments * Address comments * Flush all tasks from local lineage cache after a node failure (ray-project#4964) * Remove typing from setup.py install_requirements. (ray-project#4971) * [Java] Fix bug of `BaseID` in multi-threading case. (ray-project#4974) * [rllib] Fix DDPG example (ray-project#4973) * Upgrade CI clang-format to 6.0 (ray-project#4976) * [Core worker] add store & task provider (ray-project#4966) * Fix bugs in the a3c code template. (ray-project#4984) * Inherit Function Docstrings and other metedata (ray-project#4985) * Fix a crash when unknown worker registering to raylet (ray-project#4992) * [gRPC] Use gRPC for inter-node-manager communication (ray-project#4968)

* [rllib] Remove dependency on TensorFlow (ray-project#4764) * remove hard tf dep * add test * comment fix * fix test * Dynamic Custom Resources - create and delete resources (ray-project#3742) * Update tutorial link in doc (ray-project#4777) * [rllib] Implement learn_on_batch() in torch policy graph * Fix `ray stop` by killing raylet before plasma (ray-project#4778) * Fatal check if object store dies (ray-project#4763) * [rllib] fix clip by value issue as TF upgraded (ray-project#4697) * fix clip_by_value issue * fix typo * [autoscaler] Fix submit (ray-project#4782) * Queue tasks in the raylet in between async callbacks (ray-project#4766) * Add a SWAP TaskQueue so that we can keep track of tasks that are temporarily dequeued * Fix bug where tasks that fail to be forwarded don't appear to be local by adding them to SWAP queue * cleanups * updates * updates * [Java][Bazel] Refine auto-generated pom files (ray-project#4780) * Bump version to 0.7.0 (ray-project#4791) * [JAVA] setDefaultUncaughtExceptionHandler to log uncaught exception in user thread. (ray-project#4798) * Add WorkerUncaughtExceptionHandler * Fix * revert bazel and pom * [tune] Fix CLI test (ray-project#4801) * Fix pom file generation (ray-project#4800) * [rllib] Support continuous action distributions in IMPALA/APPO (ray-project#4771) * [rllib] TensorFlow 2 compatibility (ray-project#4802) * Change tagline in documentation and README. (ray-project#4807) * Update README.rst, index.rst, tutorial.rst and _config.yml * [tune] Support non-arg submit (ray-project#4803) * [autoscaler] rsync cluster (ray-project#4785) * [tune] Remove extra parsing functionality (ray-project#4804) * Fix Java worker log dir (ray-project#4781) * [tune] Initial track integration (ray-project#4362) Introduces a minimally invasive utility for logging experiment results. A broad requirement for this tool is that it should integrate seamlessly with Tune execution. * [rllib] [RFC] Dynamic definition of loss functions and modularization support (ray-project#4795) * dynamic graph * wip * clean up * fix * document trainer * wip * initialize the graph using a fake batch * clean up dynamic init * wip * spelling * use builder for ppo pol graph * add ppo graph * fix naming * order * docs * set class name correctly * add torch builder * add custom model support in builder * cleanup * remove underscores * fix py2 compat * Update dynamic_tf_policy_graph.py * Update tracking_dict.py * wip * rename * debug level * rename policy_graph -> policy in new classes * fix test * rename ppo tf policy * port appo too * forgot grads * default policy optimizer * make default config optional * add config to optimizer * use lr by default in optimizer * update * comments * remove optimizer * fix tuple actions support in dynamic tf graph * [rllib] Rename PolicyGraph => Policy, move from evaluation/ to policy/ (ray-project#4819) This implements some of the renames proposed in ray-project#4813 We leave behind backwards-compatibility aliases for *PolicyGraph and SampleBatch. * [Java] Dynamic resource API in Java (ray-project#4824) * Add default values for Wgym flags * Fix import * Fix issue when starting `raylet_monitor` (ray-project#4829) * Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID (ray-project#4776) * Enable BaseId. * Change TaskID and make python test pass * Remove unnecessary functions and fix test failure and change TaskID to 16 bytes. * Java code change draft * Refine * Lint * Update java/api/src/main/java/org/ray/api/id/TaskId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/BaseId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/api/src/main/java/org/ray/api/id/ObjectId.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comment * Lint * Fix SINGLE_PROCESS * Fix comments * Refine code * Refine test * Resolve conflict * Fix bug in which actor classes are not exported multiple times. (ray-project#4838) * Bump Ray master version to 0.8.0.dev0 (ray-project#4845) * Add section to bump version of master branch and cleanup release docs (ray-project#4846) * Fix import * Export remote functions when first used and also fix bug in which rem… (ray-project#4844) * Export remote functions when first used and also fix bug in which remote functions and actor classes are not exported from workers during subsequent ray sessions. * Documentation update * Fix tests. * Fix grammar * Update wheel versions in documentation to 0.8.0.dev0 and 0.7.0. (ray-project#4847) * [tune] Later expansion of local_dir (ray-project#4806) * [rllib] [RFC] Deprecate Python 2 / RLlib (ray-project#4832) * Fix a typo in kubernetes yaml (ray-project#4872) * Move global state API out of global_state object. (ray-project#4857) * Install bazel in autoscaler development configs. (ray-project#4874) * [tune] Fix up Ax Search and Examples (ray-project#4851) * update Ax for cleaner API * docs update * [rllib] Update concepts docs and add "Building Policies in Torch/TensorFlow" section (ray-project#4821) * wip * fix index * fix bugs * todo * add imports * note on get ph * note on get ph * rename to building custom algs * add rnn state info * [rllib] Fix error getting kl when simple_optimizer: True in multi-agent PPO * Replace ReturnIds with NumReturns in TaskInfo to reduce the size (ray-project#4854) * Refine TaskInfo * Fix * Add a test to print task info size * Lint * Refine * Update deps commits of opencensus to support building with bzl 0.25.x (ray-project#4862) * Update deps to support bzl 2.5.x * Fix * Upgrade arrow to latest master (ray-project#4858) * [tune] Auto-init Ray + default SearchAlg (ray-project#4815) * Bump version from 0.8.0.dev0 to 0.7.1. (ray-project#4890) * [rllib] Allow access to batches prior to postprocessing (ray-project#4871) * [rllib] Fix Multidiscrete support (ray-project#4869) * Refactor redis callback handling (ray-project#4841) * Add CallbackReply * Fix * fix linting by format.sh * Fix linting * Address comments. * Fix * Initial high-level code structure of CoreWorker. (ray-project#4875) * Drop duplicated string format (ray-project#4897) This string format is unnecessary. java_worker_options has been appended to the commandline later. * Refactor ID Serial 2: change all ID functions to `CamelCase` (ray-project#4896) * Hotfix for change of from_random to FromRandom (ray-project#4909) * [rllib] Fix documentation on custom policies (ray-project#4910) * wip * add docs * lint * todo sections * fix doc * [rllib] Allow Torch policies access to full action input dict in extra_action_out_fn (ray-project#4894) * fix torch extra out * preserve setitem * fix docs * [tune] Pretty print params json in logger.py (ray-project#4903) * [sgd] Distributed Training via PyTorch (ray-project#4797) Implements distributed SGD using distributed PyTorch. * [rllib] Rough port of DQN to build_tf_policy() pattern (ray-project#4823) * fetching objects in parallel in _get_arguments_for_execution (ray-project#4775) * [tune] Disallow setting resources_per_trial when it is already configured (ray-project#4880) * disallow it * import fix * fix example * fix test * fix tests * Update mock.py * fix * make less convoluted * fix tests * [rllib] Rename PolicyEvaluator => RolloutWorker (ray-project#4820) * Fix local cluster yaml (ray-project#4918) * [tune] Directional metrics for components (ray-project#4120) (ray-project#4915) * [Core Worker] implement ObjectInterface and add test framework (ray-project#4899) * [tune] Make PBT Quantile fraction configurable (ray-project#4912) * Better organize ray_common module (ray-project#4898) * Fix error * [tune] Add requirements-dev.txt and update docs for contributing (ray-project#4925) * Add requirements-dev.txt and update docs. * Update doc/source/tune-contrib.rst Co-Authored-By: Richard Liaw <rliaw@berkeley.edu> * Unpin everything except for yapf. * Fix compute actions return value * Bump version from 0.7.1 to 0.8.0.dev1. (ray-project#4937) * Update version number in documentation after release 0.7.0 -> 0.7.1 and 0.8.0.dev0 -> 0.8.0.dev1. (ray-project#4941) * [doc] Update developer docs with bazel instructions (ray-project#4944) * [C++] Add hash table to Redis-Module (ray-project#4911) * Flush lineage cache on task submission instead of execution (ray-project#4942) * [rllib] Add docs on how to use TF eager execution (ray-project#4927) * [rllib] Port remainder of algorithms to build_trainer() pattern (ray-project#4920) * Fix resource bookkeeping bug with acquiring unknown resource. (ray-project#4945) * Update aws keys for uploading wheels to s3. (ray-project#4948) * Upload wheels on Travis to branchname/commit_id. (ray-project#4949) * [Java] Fix serializing issues of `RaySerializer` (ray-project#4887) * Fix * Address comment. * fix (ray-project#4950) * [Java] Add inner class `Builder` to build call options. (ray-project#4956) * Add Builder class * format * Refactor by IDE * Remove uncessary dependency * Make release stress tests work and improve them. (ray-project#4955) * Use proper session directory for debug_string.txt (ray-project#4960) * [core] Use int64_t instead of int to keep track of fractional resources (ray-project#4959) * [core worker] add task submission & execution interface (ray-project#4922) * [sgd] Add non-distributed PyTorch runner (ray-project#4933) * Add non-distributed PyTorch runner * use dist.is_available() instead of checking OS * Nicer exception * Fix bug in choosing port * Refactor some code * Address comments * Address comments * Flush all tasks from local lineage cache after a node failure (ray-project#4964) * Remove typing from setup.py install_requirements. (ray-project#4971) * [Java] Fix bug of `BaseID` in multi-threading case. (ray-project#4974) * [rllib] Fix DDPG example (ray-project#4973) * Upgrade CI clang-format to 6.0 (ray-project#4976) * [Core worker] add store & task provider (ray-project#4966) * Fix bugs in the a3c code template. (ray-project#4984) * Inherit Function Docstrings and other metedata (ray-project#4985) * Fix a crash when unknown worker registering to raylet (ray-project#4992) * [gRPC] Use gRPC for inter-node-manager communication (ray-project#4968) * Fix Java CI failure (ray-project#4995) * fix handling of non-integral timeout values in signal.receive (ray-project#5002) * temp fix for build (ray-project#5006) * [tune] Tutorial UX Changes (ray-project#4990) * add integration, iris, ASHA, recursive changes, set reuse_actors=True, and enable Analysis as a return object * docstring * fix up example * fix * cleanup tests * experiment analysis * Fix valgrind build by installing new version of valgrind (ray-project#5008) * Fix no cpus test (ray-project#5009) * Fix tensorflow-1.14 installation in jenkins (ray-project#5007) * Add dynamic worker options for worker command. (ray-project#4970) * Add fields for fbs * WIP * Fix complition errors * Add java part * FIx * Fix * Fix * Fix lint * Refine API * address comments and add test * Fix * Address comment. * Address comments. * Fix linting * Refine * Fix lint * WIP: address comment. * Fix java * Fix py * Refin * Fix * Fix * Fix linting * Fix lint * Address comments * WIP * Fix * Fix * minor refine * Fix lint * Fix raylet test. * Fix lint * Update src/ray/raylet/worker_pool.h Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Update java/runtime/src/main/java/org/ray/runtime/AbstractRayRuntime.java Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comments. * Address comments. * Fix test. * Update src/ray/raylet/worker_pool.h Co-Authored-By: Hao Chen <chenh1024@gmail.com> * Address comments. * Address comments. * Fix * Fix lint * Fix lint * Fix * Address comments. * Fix linting * [docs] docs for running Tensorboard without sudo (ray-project#5015) * Instructions for running Tensorboard without sudo When we run Tensorboard to visualize the results of Ray outputs on multi-user clusters where we don't have sudo access, such as RISE clusters, a few commands need to first be run to make sure tensorboard can edit the tmp directory. This is a pretty common usecase so I figured we may as well put it in the documentation for Tune. * Update tune-usage.rst * [ci] Change Jenkins to py3 (ray-project#5022) * conda3 * integration * add nevergrad, remotedata * pytest 0.3.1 * otherdockers * setup * tune * [gRPC] Migrate gcs data structures to protobuf (ray-project#5024) * [rllib] Add QMIX mixer parameters to optimizer param list (ray-project#5014) * add mixer params * Update qmix_policy.py * [grpc] refactor rpc server to support multiple io services (ray-project#5023) * [rllib] Give error if sample_async is used with pytorch for A3C (ray-project#5000) * give error if sample_async is used with pytorch * update * Update a3c.py * [tune] Update MNIST Example (ray-project#4991) * Add entropy coeff schedule * Revert "Merge with ray master" This reverts commit 108bfa2, reversing changes made to 2e0eec9. * Revert "Revert "Merge with ray master"" This reverts commit 92c0f88. * Remove entropy decay stuff

This was referenced Jan 10, 2019

[WIP] Supporting dynamic custom resources #3439

Closed

Alternative solution for actor affinity and anti-affinity #3432

Closed

romilbhardwaj force-pushed the dynamic-res branch from 1811b04 to 825ad7a Compare February 5, 2019 08:16

romilbhardwaj force-pushed the dynamic-res branch from ebf83c5 to c65524b Compare February 14, 2019 22:55

romilbhardwaj requested review from robertnishihara, stephanie-wang, atumanov, raulchen and pcmoritz February 16, 2019 03:14

romilbhardwaj changed the title ~~[WIP] Dynamic Custom Resources - create and delete resources~~ Dynamic Custom Resources - create and delete resources Feb 16, 2019

ray-project deleted a comment from AmplabJenkins Feb 16, 2019

cpp linting

737f5eb

Fix GCS tests

917e97c

romilbhardwaj added 2 commits May 1, 2019 14:26

Fix pylinting

137e57a

Fix py2 compatibility

1c1080c

raulchen approved these changes May 8, 2019

View reviewed changes

raulchen requested changes May 8, 2019

View reviewed changes

guoyuhong reviewed May 9, 2019

View reviewed changes

romilbhardwaj added 3 commits May 9, 2019 16:38

Resolve comments

c446548

Java fixes

b22c884

fix python2 compatibility

4baa134

raulchen reviewed May 10, 2019

View reviewed changes

java/BUILD.bazel Outdated Show resolved Hide resolved

Undo changes to BUILD.bazel

ae13704

raulchen approved these changes May 10, 2019

View reviewed changes

Remove sleeps

0dde285

Linting

feee8c7

guoyuhong approved these changes May 10, 2019

View reviewed changes

raulchen merged commit 004440f into ray-project:master May 11, 2019

kfstorm reviewed Jun 14, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dynamic Custom Resources - create and delete resources #3742

Dynamic Custom Resources - create and delete resources #3742

romilbhardwaj commented Jan 10, 2019 •

edited

Loading

AmplabJenkins commented Jan 10, 2019

robertnishihara commented Jan 12, 2019

raulchen commented Jan 13, 2019

raulchen commented Jan 13, 2019

AmplabJenkins commented Jan 15, 2019

romilbhardwaj commented Jan 15, 2019 •

edited

Loading

AmplabJenkins commented Jan 15, 2019

AmplabJenkins commented Jan 15, 2019

raulchen commented Jan 16, 2019

jovany-wang commented Jan 25, 2019

robertnishihara commented Feb 5, 2019 •

edited

Loading

AmplabJenkins commented Feb 8, 2019

virtualluke commented Feb 13, 2019

romilbhardwaj commented Feb 14, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 2, 2019

raulchen left a comment

raulchen May 8, 2019

raulchen May 8, 2019

raulchen left a comment

guoyuhong left a comment

guoyuhong May 9, 2019

AmplabJenkins commented May 10, 2019

AmplabJenkins commented May 10, 2019

AmplabJenkins commented May 10, 2019

kfstorm Jun 14, 2019

		assert successful # The task completed


		def test_dynamic_res_updation_clientid(ray_start_cluster):

Dynamic Custom Resources - create and delete resources #3742

Dynamic Custom Resources - create and delete resources #3742

Conversation

romilbhardwaj commented Jan 10, 2019 • edited Loading

What do these changes do?

Related issue number

AmplabJenkins commented Jan 10, 2019

robertnishihara commented Jan 12, 2019

raulchen commented Jan 13, 2019

raulchen commented Jan 13, 2019

AmplabJenkins commented Jan 15, 2019

romilbhardwaj commented Jan 15, 2019 • edited Loading

AmplabJenkins commented Jan 15, 2019

AmplabJenkins commented Jan 15, 2019

raulchen commented Jan 16, 2019

jovany-wang commented Jan 25, 2019

robertnishihara commented Feb 5, 2019 • edited Loading

AmplabJenkins commented Feb 8, 2019

virtualluke commented Feb 13, 2019

romilbhardwaj commented Feb 14, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented Feb 16, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 1, 2019

AmplabJenkins commented May 2, 2019

raulchen left a comment

Choose a reason for hiding this comment

raulchen May 8, 2019

Choose a reason for hiding this comment

raulchen May 8, 2019

Choose a reason for hiding this comment

raulchen left a comment

Choose a reason for hiding this comment

guoyuhong left a comment

Choose a reason for hiding this comment

guoyuhong May 9, 2019

Choose a reason for hiding this comment

AmplabJenkins commented May 10, 2019

AmplabJenkins commented May 10, 2019

AmplabJenkins commented May 10, 2019

kfstorm Jun 14, 2019

Choose a reason for hiding this comment

romilbhardwaj commented Jan 10, 2019 •

edited

Loading

romilbhardwaj commented Jan 15, 2019 •

edited

Loading

robertnishihara commented Feb 5, 2019 •

edited

Loading