General attribute-based heterogeneity support with hard and soft constraints #248

atumanov · 2017-02-06T21:12:16Z

modifications to the global and local scheduler to provide attribute-based resource accounting
support for hard capacity constraints
support for soft dynamic resource capacity constraints, including transfer-cost awareness, in the global scheduler

…nd robin

pcmoritz · 2017-02-07T04:45:12Z

python/global_scheduler/test/test.py

-        redis_address=redis_address)
-    # Connect to the scheduler.
-    self.photon_client = photon.PhotonClient(local_scheduler_name)
+    self.plasma_store_p2 = []


this name is a little unusual, want to rename to self.plasma_stores? (similar for plasma_store_p3 and 4)

Could also put all the processes in one list?

hmm, actually, I'd rather keep them separate, because I'd prefer to kill all photons, followed by all plasma managers, followed by all plasma stores.

pcmoritz · 2017-02-07T04:46:13Z

python/global_scheduler/test/test.py

    redis_path = os.path.join(os.path.abspath(os.path.dirname(__file__)), "../../core/src/common/thirdparty/redis/src/redis-server")
    redis_module = os.path.join(os.path.dirname(os.path.abspath(__file__)), "../../core/src/common/redis_module/libray_redis_module.so")
    assert os.path.isfile(redis_path)
    assert os.path.isfile(redis_module)
    node_ip_address = "127.0.0.1"
+    #redis_port = 6379 #default Redis port


remove this

pcmoritz · 2017-02-07T04:47:48Z

python/global_scheduler/test/test.py

+      plasma_store_name, p2 = plasma.start_plasma_store()
+      self.plasma_store_p2.append(p2)
+      # Start the Plasma manager.
+      # Assumption: Plasma manager name and port are randomly generated by the plasma module.


Can get rid of "Assumption" here

I had to check to make sure this is indeed the case, because we need different (name, port) pairs generated for the multinode test case. If the generated (name, port) pair were deterministic, I would've had to make changes inside the plasma module. I think explicitly stating this post-condition is useful.

pcmoritz · 2017-02-07T05:02:56Z

src/global_scheduler/global_scheduler.h

@@ -39,10 +39,12 @@ typedef struct {
  UT_hash_handle hh;
 } scheduler_object_info;

+/* plasma_manager ip:port -> photon_db_client_id association entry. */


Can we put the documentation into the same form as everywhere else, i.e. full sentences ("Association of foo to bar" is ok), doxygen style (i.e. /** instead of /*) and for members preceding the member and not following it.

sure, maybe we should add doxygen generation to the build, so it explicitly fails the build if the comments are not doxygen compliant. Since this is a header file (for which we expect to produce doxygen documentation), I agree with your comment placement suggestion.

robertnishihara · 2017-02-07T05:56:16Z

python/global_scheduler/test/test.py

    db_client_id = self.get_plasma_manager_id()
    assert(db_client_id != None)
    assert(db_client_id.startswith(b"CL:"))
    db_client_id = db_client_id[len(b"CL:"):] # Remove the CL: prefix.
-
+  


white space

I leave all white space-related cleanup until the very end and run all the code through the linter that formats it for me.

robertnishihara · 2017-02-07T05:56:21Z

python/global_scheduler/test/test.py

-    self.assertEqual(len(self.redis_client.keys("{}*".format(DB_CLIENT_PREFIX))), 3)
-
+    self.assertEqual(len(self.redis_client.keys("{}*".format(DB_CLIENT_PREFIX))), 2*NUM_CLUSTER_NODES+1)
+  


white space

robertnishihara · 2017-02-07T05:56:53Z

python/global_scheduler/test/test.py

    num_return_vals = [0, 1, 2, 3, 5, 10]
    # There should not be anything else in Redis yet.
-    self.assertEqual(len(self.redis_client.keys("*")), 3)
+    self.assertEqual(len(self.redis_client.keys("*")), 2*NUM_CLUSTER_NODES+1)


2 * NUM_CLUSTER_NODES + 1

robertnishihara · 2017-02-07T05:58:37Z

python/global_scheduler/test/test.py

+          plasma_manager_name=plasma_manager_name,
+          plasma_address=plasma_address,
+          redis_address=redis_address,
+          static_resource_list=[str(DEFAULT_NUM_CPUS), str(DEFAULT_NUM_GPUS)])


I think it makes more sense to use ints here and turn them into strings inside of photon.start_local_scheduler

robertnishihara · 2017-02-07T05:59:31Z

python/ray/services.py

@@ -262,7 +262,8 @@ def start_global_scheduler(redis_address, cleanup=True, redirect_output=False):

 def start_local_scheduler(redis_address, node_ip_address, plasma_store_name,
                          plasma_manager_name, worker_path, plasma_address=None,
-                          cleanup=True, redirect_output=False):
+                          cleanup=True, redirect_output=False,
+                          static_resource_list=None):


Document this argument

robertnishihara · 2017-02-07T06:28:48Z

src/global_scheduler/global_scheduler.c

        strlen(plasma_photon_entry->aux_address), plasma_photon_entry);

+    /* Add photon_db_client_id -> plasma_manager ip:port association to state.*/
+    HASH_ADD(photon_plasma_hh, state->photon_plasma_map,
+             photon_db_client_id, /* Key is the field name of entry struct. */


We've been always putting comments on their own lines and not on separate lines.

I know, but this is a rather inconvenient stylistic choice. When a comment is on the same line as the code it comments on, there's no ambiguity which line of code it corresponds to. When the comment is on its own line, it could correspond to the line before or the line after. It's only by convention that you know it's the latter.
It makes sense for header files, when we expect doxygen to generate API documentation, but why have the same constraint in the rest of the code, where comments are rather for the developers reading the code?

robertnishihara · 2017-02-07T06:29:05Z

src/global_scheduler/global_scheduler.c

        strlen(plasma_photon_entry->aux_address), plasma_photon_entry);

+    /* Add photon_db_client_id -> plasma_manager ip:port association to state.*/


state.*/ -> state. */

robertnishihara · 2017-02-07T06:29:56Z

src/global_scheduler/global_scheduler_algorithm.c

@@ -10,6 +10,15 @@ global_scheduler_policy_state *init_global_scheduler_policy(void) {
  global_scheduler_policy_state *policy_state =
      malloc(sizeof(global_scheduler_policy_state));
  policy_state->round_robin_index = 0;
+
+  int num_weight_elem = sizeof(policy_state->resource_attribute_weight)/sizeof(double);


spaces around /

Also in = 1.0/num_weight_elem;

robertnishihara · 2017-02-07T06:31:06Z

src/global_scheduler/global_scheduler_algorithm.c

+
+  for (i = policy_state->round_robin_index;
+       !task_satisfied && num_retries >= 0;
+       i = (i+1) % utarray_len(state->local_schedulers)) {


robertnishihara · 2017-02-07T06:39:11Z

src/common/task.h

+  MAX_RESOURCE_INDEX
+} resource_vector_index;
+
+#define DEFAULT_NUM_CPUS 16


I'm not sure if we should have defaults. Maybe the arguments should be required to be passed in to the local scheduler?

robertnishihara · 2017-02-07T06:53:01Z

src/photon/photon_scheduler.c

+         * so we set the pointer to NULL so it is not used. */
+        worker->task_in_progress = NULL;
+      }
+      if (worker->task_in_progress) {


It doesn't change the code, but maybe making if (worker->task_in_progress) { into else if would be a bit easier to understand

robertnishihara · 2017-02-07T07:52:15Z

src/global_scheduler/global_scheduler_algorithm.c

+  CHECKM(task_spec != NULL,
+         "task wait handler encounted a task with NULL spec");
+#if (RAY_COMMON_LOG_LEVEL <= RAY_COMMON_DEBUG)
+  char buf[256];


Using fixed size buffers seems a bit dangerous. We could use UT_string to get around this.

actually, fixed size string buffers are OK, as long as I use snprintf(), which is guaranteed to not write more than the specified size. This is for debugging only. I could remove this whole #if #endif block.

stephanie-wang · 2017-02-07T19:20:36Z

python/global_scheduler/test/test.py

-        redis_address=redis_address)
-    # Connect to the scheduler.
-    self.photon_client = photon.PhotonClient(local_scheduler_name)
+    self.plasma_store_p2 = []


Could also put all the processes in one list?

stephanie-wang · 2017-02-07T19:24:16Z

src/common/task.c

@@ -259,6 +265,10 @@ int64_t task_args_add_val(task_spec *spec, uint8_t *data, int64_t length) {
  return spec->arg_index++;
 }

+void task_add_required_resource(task_spec *spec, int64_t resource_index, double value) {
+  spec->required_resources[resource_index] = value;


Should probably do some error checking on value (e.g., PyFloat_AsDouble can fail and return -1). Also, are there required resources that wouldn't have an integer value?

The checking should probably be in the Python code then.

stephanie-wang · 2017-02-07T19:33:53Z

src/global_scheduler/global_scheduler_algorithm.c

+
+  if (task_satisfied) {
+    /* Update next index to try and assign the task. */
+    policy_state->round_robin_index = i; /* i was advanced. */


It's not a big deal, but if the above goes to the last retry (num_retries = -1) and the task is satisfied, I think round_robin_index will be set to the same index as the one the task is assigned to, instead of the one after.

i'm pretty sure that i is always advanced, as long as we take at least one pass through the body of the for loop.

stephanie-wang · 2017-02-07T19:34:41Z

src/global_scheduler/global_scheduler_algorithm.c

+  int num_retries = NUM_RETRIES;
+  bool task_satisfied = false;
+
+  for (i = policy_state->round_robin_index;


ls->info.static_resources doesn't change between iterations, right? Can we limit the number of rounds to the minimum of the number of local schedulers and NUM_RETRIES?

…apacity (until next photon heartbeat)

… into hetergen-sched-rebase

robertnishihara · 2017-02-08T07:58:19Z

python/ray/services.py

@@ -18,6 +18,7 @@
 import photon
 import plasma
 import global_scheduler
+from IPython.utils.sysinfo import num_cpus


Let's not import this. We can't assume that people have IPython.

robertnishihara · 2017-02-08T08:06:41Z

Note, this error message is out of date. We should fix it before merging.

AssertionError: The @ray.remote decorator must be applied either with no arguments and no parentheses, for example '@ray.remote', or it must be applied with only the argument num_return_vals, like '@ray.remote(num_return_vals=2)'.

robertnishihara · 2017-02-08T19:53:48Z

src/global_scheduler/global_scheduler.c

+    /** Subtract task's resource from the cached dynamic resource capacity for
+     *  this local scheduler. This will be overwritten on the next heartbeat. */
+    local_scheduler->info.dynamic_resources[i] =
+        MAX(0, local_scheduler->info.dynamic_resources[i] -


We shouldn't take a MAX. It should be allowed to go negative (otherwise, once all nodes hit 0 the global scheduler won't know how to distinguish between them even if way more tasks have been assigned to one of them).

I'm worried this may have unexpected effects on the score. Just letting it be negative doesn't have the effect you'd want on the score. E.g. dyn_cpu_node1 == -2, dyn_cpu_node2 = -1 and requested cpu=1. Node1 would cause less negative delta in the score than node2, and we'd want it to be reversed. dynfrac_cpu_node1 = -0.5 and dynfrac_cpu_node2 = -1. So node2 appears worse, while it should be better.

robertnishihara · 2017-02-08T21:19:03Z

src/global_scheduler/global_scheduler_algorithm.c

-    handle_task_round_robin(state, policy_state, task);
-    return;
-  }
+    /* This node satisfies the hard capacity constraint. Calculate its score. */


All of the logic about computing a "score" should be separated out as a different function so we can easily change the computation of the score without changing how the scheduling policy uses the score (or so we can use the score in another scheduling policy).

pcmoritz · 2017-02-08T22:06:11Z

python/global_scheduler/test/test.py

-    self.p3.kill()
-    self.p4.kill()
+    # kill photons, plasma managers, and plasma stores.
+    map(subprocess.Popen.kill, self.local_scheduler_pids) #kill photon


add spaces before # at the end of line

you mean after?

pcmoritz · 2017-02-08T22:11:00Z

python/photon/photon_services.py

@@ -71,6 +73,21 @@ def start_local_scheduler(plasma_store_name, plasma_manager_name=None,
    command += ["-r", redis_address]
  if plasma_address is not None:
    command += ["-a", plasma_address]
+  # We want to be able to support indepdently setting capacity for each of the supported resource


independently

pcmoritz · 2017-02-08T22:26:19Z

src/common/task.h

+ * @param value Value for the resource. This can be a quantity of resources
+ *        this task needs for example or it can be a value for an
+ *        attribute this task requires.
+ * @return How many of this resource the task needs to execute.


replace by

@return Void.

documentation updated.

stephanie-wang · 2017-02-08T23:00:13Z

python/ray/services.py

@@ -409,11 +415,22 @@ def start_ray_processes(address_info=None,
      start a global scheduler process.
    include_redis (bool): If include_redis is True, then start a Redis server
      process.
+    num_cpus: A list of length num_local_schedulers containing the number of


We're probably going to need to pass in more resources in the future. It may be a good idea to use a Python namedtuple to define a ResourceList type, so that we can pass in a single argument for all the resources, and not have to remember the ordering of such an argument.

we'll generalize the frontend in a separate PR. The initial plan was to just add support for the GPUs in the frontend. We made the backend general already. It'll be easy to make the frontend general when we get more use cases.

stephanie-wang · 2017-02-08T23:04:20Z

src/common/lib/python/common_extension.c

+  } else {
+    for (int i = 0; i < MAX_RESOURCE_INDEX; ++i) {
+      task_spec_add_required_resource(self->spec, i,
+                                      i == CPU_RESOURCE_INDEX ? 1.0 : 0.0);


This code isn't very generalizable to other resources. I would either leave a comment explaining that these are the default values for these particular indices, or make a static double[] containing the default values.

why is it not generalizable? The purpose of this code is to just set the number of requested cores to one and leave everything else zero. This is pretty general, IMHO. It does assume that the default is zero for everything else.

stephanie-wang · 2017-02-08T23:04:46Z

src/common/task.h

+ *        attribute this task requires.
+ * @return How many of this resource the task needs to execute.
+ */
+void task_spec_add_required_resource(task_spec *spec,


Maybe call this task_spec_set_required_resource (and then get for the one below).

yeah, I like the symmetry, will do as a separate commit (global replace)

stephanie-wang · 2017-02-08T23:10:14Z

src/photon/photon_scheduler.c

-      free_task(worker->task_in_progress);
-      worker->task_in_progress = NULL;
+    task *task_in_progress = worker->task_in_progress;
+    task_spec *spec =


Can you move this line into the if-block below? This is just personal preference, but I always feel that ternary operators are harder to read.

I love ternary operators. In fact I miss them in python dearly. But sure, after multiple rounds of iterations on this code, it no longer makes sense to have this spec outside of the if that follows.

I took a full pass through this whole case statement and minimized it.

stephanie-wang · 2017-02-08T23:10:53Z

src/photon/photon_scheduler.c

+      }
+      print_resource_info(state, spec);
+      /* If we're connected to Redis, update tables. */
+      if (state->db != NULL && task_in_progress != NULL) {


Remove the task_in_progress != NULL check.

stephanie-wang · 2017-02-08T23:11:20Z

src/photon/photon_scheduler.c

+         * task_in_progress,
+         * so we set the pointer to NULL so it is not used. */
+        worker->task_in_progress = NULL;
+      } else if (worker->task_in_progress) {


Make this just else.

stephanie-wang · 2017-02-08T23:14:55Z

test/runtest.py

+    def f(n):
+      time.sleep(n)
+
+    start_time = time.time()


Can you document these blocks? Also, set a variable equal to 0.5 instead of referencing it by value.

stephanie-wang · 2017-02-08T23:17:16Z

src/photon/photon_scheduler.h

@@ -7,6 +7,9 @@
 /* The duration between local scheduler heartbeats. */
 #define LOCAL_SCHEDULER_HEARTBEAT_TIMEOUT_MILLISECONDS 100

+#define DEFAULT_NUM_CPUS INT16_MAX


Can we get the number of cores, instead of setting this to a static value?

this has been discussed and the conclusion is to set default number of cores to infinity if not specified. This makes the behavior backward compatible.
In most cases, we start photon from python, and we automatically figure out the number of hw threads before spawning the photon process.
The expectation is for the resources to be specified when the local scheduler is started.

I think it makes sense to use this as the default if nothing is passed in.

stephanie-wang · 2017-02-08T23:20:47Z

src/common/lib/python/common_extension.c

@@ -317,6 +319,20 @@ static int PyTask_init(PyTask *self, PyObject *args, PyObject *kwds) {
    }
  }
  utarray_free(val_repr_ptrs);
+  /* Set the resource vector of the task. */
+  if (resource_vector != NULL) {


I wonder if we should be checking that the resource values specified by the caller are above some minimum value. If we use 1.0 as the default number of CPUs instead of the minimum, we could specify a task as using 0 CPUs, and then we won't track the dynamic resources properly in the schedulers. Could lead to unfair task assignments, where one task is starved just because it specified a more conservative resource requirement, whereas another task specifies 0 CPUs but actually uses n.

I suppose we could check if requested CPU is < 1, but I know for a fact that in the Google cluster it is entirely possible to request fractional cores. The workload may be entirely memory bound, for instance, in which case it doesn't need the whole core to be dedicated to it. Fairness of these allocations would have to be a feature we add on the backend when we actually enable attribution of tasks (and , therefore, resources) to users (or drivers)

pcmoritz · 2017-02-08T23:42:41Z

test/runtest.py

+    ray.worker.global_worker.run_function_on_all_workers(lambda worker_info: sys.path.append(worker_info["counter"]))
+    @ray.remote(num_cpus=0)
+    def get_worker_id():
+      time.sleep(1)


why do we need the sleep here?

answer: to make sure tasks are scheduled on different workers

To make sure the tasks all run on different workers.

atumanov · 2017-02-09T00:36:59Z

src/photon/photon_scheduler.h

@@ -7,6 +7,9 @@
 /* The duration between local scheduler heartbeats. */
 #define LOCAL_SCHEDULER_HEARTBEAT_TIMEOUT_MILLISECONDS 100

+#define DEFAULT_NUM_CPUS INT16_MAX


this has been discussed and the conclusion is to set default number of cores to infinity if not specified. This makes the behavior backward compatible.
In most cases, we start photon from python, and we automatically figure out the number of hw threads before spawning the photon process.
The expectation is for the resources to be specified when the local scheduler is started.

atumanov · 2017-02-09T00:38:42Z

src/photon/photon_scheduler.h

@@ -7,6 +7,9 @@
 /* The duration between local scheduler heartbeats. */
 #define LOCAL_SCHEDULER_HEARTBEAT_TIMEOUT_MILLISECONDS 100

+#define DEFAULT_NUM_CPUS INT16_MAX
+#define DEFAULT_NUM_GPUS 0


same here, the default was set to be backward compatible and the expectation is that the local scheduler is started with resource config specified. In the long run, we envision that the local scheduler will be reading a config file to get these values.

atumanov · 2017-02-09T00:40:16Z

python/global_scheduler/test/test.py

-    self.p3.kill()
-    self.p4.kill()
+    # kill photons, plasma managers, and plasma stores.
+    map(subprocess.Popen.kill, self.local_scheduler_pids) #kill photon


you mean after?

atumanov · 2017-02-09T00:46:33Z

python/ray/services.py

@@ -409,11 +415,22 @@ def start_ray_processes(address_info=None,
      start a global scheduler process.
    include_redis (bool): If include_redis is True, then start a Redis server
      process.
+    num_cpus: A list of length num_local_schedulers containing the number of


we'll generalize the frontend in a separate PR. The initial plan was to just add support for the GPUs in the frontend. We made the backend general already. It'll be easy to make the frontend general when we get more use cases.

atumanov · 2017-02-09T01:08:11Z

src/common/lib/python/common_extension.c

@@ -317,6 +319,20 @@ static int PyTask_init(PyTask *self, PyObject *args, PyObject *kwds) {
    }
  }
  utarray_free(val_repr_ptrs);
+  /* Set the resource vector of the task. */
+  if (resource_vector != NULL) {


I suppose we could check if requested CPU is < 1, but I know for a fact that in the Google cluster it is entirely possible to request fractional cores. The workload may be entirely memory bound, for instance, in which case it doesn't need the whole core to be dedicated to it. Fairness of these allocations would have to be a feature we add on the backend when we actually enable attribution of tasks (and , therefore, resources) to users (or drivers)

atumanov · 2017-02-09T01:12:41Z

src/common/task.h

+ * @param value Value for the resource. This can be a quantity of resources
+ *        this task needs for example or it can be a value for an
+ *        attribute this task requires.
+ * @return How many of this resource the task needs to execute.


documentation updated.

atumanov · 2017-02-09T01:13:56Z

src/common/task.h

+ *        attribute this task requires.
+ * @return How many of this resource the task needs to execute.
+ */
+void task_spec_add_required_resource(task_spec *spec,


yeah, I like the symmetry, will do as a separate commit (global replace)

atumanov · 2017-02-09T01:18:59Z

src/global_scheduler/global_scheduler.c

+    /** Subtract task's resource from the cached dynamic resource capacity for
+     *  this local scheduler. This will be overwritten on the next heartbeat. */
+    local_scheduler->info.dynamic_resources[i] =
+        MAX(0, local_scheduler->info.dynamic_resources[i] -


I'm worried this may have unexpected effects on the score. Just letting it be negative doesn't have the effect you'd want on the score. E.g. dyn_cpu_node1 == -2, dyn_cpu_node2 = -1 and requested cpu=1. Node1 would cause less negative delta in the score than node2, and we'd want it to be reversed. dynfrac_cpu_node1 = -0.5 and dynfrac_cpu_node2 = -1. So node2 appears worse, while it should be better.

atumanov · 2017-02-09T01:19:54Z

src/global_scheduler/global_scheduler_algorithm.c

+
+  if (task_satisfied) {
+    /* Update next index to try and assign the task. */
+    policy_state->round_robin_index = i; /* i was advanced. */


i'm pretty sure that i is always advanced, as long as we take at least one pass through the body of the for loop.

atumanov · 2017-02-09T01:22:42Z

src/photon/photon_scheduler.c

-      free_task(worker->task_in_progress);
-      worker->task_in_progress = NULL;
+    task *task_in_progress = worker->task_in_progress;
+    task_spec *spec =


I love ternary operators. In fact I miss them in python dearly. But sure, after multiple rounds of iterations on this code, it no longer makes sense to have this spec outside of the if that follows.

I took a full pass through this whole case statement and minimized it.

robertnishihara · 2017-02-09T02:06:06Z

I ran into the following problem.

import ray
ray.worker._init(num_local_schedulers=2, num_workers=10, start_ray_local=True)

@ray.remote
def f():
  return ray.worker.global_worker.plasma_client.store_socket_name

set(ray.get([f.remote() for _ in range(1000)]))

The last line prints {'/tmp/plasma_store13623057'}, which suggests that tasks are all being assigned locally.

…hetergen-sched-rebase

…first heartbeat

atumanov added 2 commits February 6, 2017 12:32

attribute-based heterogeneity-awareness in global scheduler and photon

c71830a

minor post-rebase fix

05c3ca6

atumanov mentioned this pull request Feb 6, 2017

[WIP] General attribute-based heterogeneity support with hard and soft constraints #239

Closed

atumanov added 2 commits February 6, 2017 20:08

photon: enforce dynamic capacity constraint on task dispatch

45b3439

globalsched: cap the number of times we try to schedule a task in rou…

6c3e59c

…nd robin

pcmoritz reviewed Feb 7, 2017

View reviewed changes

robertnishihara reviewed Feb 7, 2017

View reviewed changes

atumanov self-assigned this Feb 7, 2017

stephanie-wang reviewed Feb 7, 2017

View reviewed changes

atumanov and others added 5 commits February 7, 2017 20:33

propagating ability to specify resource capacity to ray.init

6aaad7d

adding resources to remote function export and fetch/register

930042c

globalsched: remove unused functions; update cached photon resource c…

319126b

…apacity (until next photon heartbeat)

Add some integration tests.

39b3a56

Merge branch 'hetergen-sched-rebase' of https://github.com/atumanov/ray…

6d7ba8c

… into hetergen-sched-rebase

robertnishihara reviewed Feb 8, 2017

View reviewed changes

atumanov added 8 commits February 8, 2017 00:18

globalsched: cleanup + factor out constraint checking

5a98f3a

lots of style

78fde2f

task_spec_required_resource: global refactor

690cda7

clang format

d5bf077

clang format + comment update in photon

40de293

clang format photon comment

93692b3

valgrind

5954219

reduce verbosity for Travis

d751e48

robertnishihara reviewed Feb 8, 2017

View reviewed changes

pcmoritz reviewed Feb 8, 2017

View reviewed changes

pcmoritz mentioned this pull request Feb 8, 2017

[WIP] First version of heterogenous scheduling #213

Closed

pcmoritz reviewed Feb 8, 2017

View reviewed changes

stephanie-wang reviewed Feb 8, 2017

View reviewed changes

pcmoritz reviewed Feb 8, 2017

View reviewed changes

atumanov commented Feb 9, 2017

View reviewed changes

robertnishihara and others added 4 commits February 8, 2017 18:37

Add test for scheduler load balancing.

8cb035c

addressing comments

1e6c0e9

refactoring global scheduler algorithm

d54d8ac

Minor cleanups.

a1272c1

robertnishihara force-pushed the hetergen-sched-rebase branch from b3008ce to a1272c1 Compare February 9, 2017 06:26

robertnishihara and others added 10 commits February 8, 2017 22:48

Linting.

5867a75

Fix array_test.py and linting.

1e756f5

valgrind fix for photon tests

76543e6

Merge branch 'hetergen-sched-rebase' of github.com:atumanov/ray into …

1e3ddf4

…hetergen-sched-rebase

Attempt to fix stress tests.

087502c

fix hashmap free

4b7afd4

fix hashmap free comment

5a34744

memset photon resource vectors to 0 in case they get used before the …

add89f3

…first heartbeat

More whitespace changes.

5fdc85a

Undo whitespace error I introduced.

c7f800a

robertnishihara merged commit dfb6107 into ray-project:master Feb 9, 2017

robertnishihara deleted the hetergen-sched-rebase branch February 9, 2017 09:35

		self.assertEqual(len(self.redis_client.keys("{}*".format(DB_CLIENT_PREFIX))), 3)

		self.assertEqual(len(self.redis_client.keys("{}".format(DB_CLIENT_PREFIX))), 2NUM_CLUSTER_NODES+1)

		strlen(plasma_photon_entry->aux_address), plasma_photon_entry);

		/* Add photon_db_client_id -> plasma_manager ip:port association to state.*/

General attribute-based heterogeneity support with hard and soft constraints #248

General attribute-based heterogeneity support with hard and soft constraints #248

Conversation

atumanov commented Feb 6, 2017

pcmoritz Feb 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz Feb 7, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

robertnishihara commented Feb 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz Feb 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz Feb 7, 2017 •

edited

Loading

pcmoritz Feb 7, 2017 •

edited

Loading

pcmoritz Feb 8, 2017 •

edited

Loading