Fix releasing CPUs incorrectly when actor creation task blocked.

[jovany-wang]

The following test can't pass before this PR:

    ray.init(num_cpus=2)

     @ray.remote(num_cpus=1)
    def get_100():
        import time
        time.sleep(2)
        return 100

     @ray.remote(num_cpus=1)
    class A(object):
        def __init__(self):
            self.num = ray.get(get_100.remote())

         def get_num(self):
            return self.num

     a = A.remote()
    assert 100 == ray.get(a.get_num.remote())
    assert 1 == ray.available_resources()["CPU"]

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-Perf-Integration-PRB/1816/
Test FAILed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15642/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15643/
Test PASSed.

[raulchen]

@romilbhardwaj I think this bug was introduced by dynamic resource. Can you take a look?

[raulchen]

@robertnishihara do you know why we need this local_available_resources_ while we already have cluster_resource_map_? why not just use cluster_resource_map_[local_node_id]?

[romilbhardwaj]

Thanks for submitting this PR. The change in node_manager.cc makes sense. However, I'm a bit confused about the test - please see my comment there.

[romilbhardwaj]

@robertnishihara do you know why we need this local_available_resources_ while we already have cluster_resource_map_? why not just use cluster_resource_map_[local_node_id]?

I think cluster_resource_map_ is a simple map of resource type and quantities used for making scheduling decisions, while local_available_resources_ handles the resource id allocation and releases at the local raylet and is used for resource id bookkeeping. Maybe we could store resource ids in cluster_resource_map_ and use a single structure for both scheduling and bookkeeping.. @robertnishihara would be able to provide better insight.

[jovany-wang]

@robertnishihara do you know why we need this local_available_resources_ while we already have cluster_resource_map_? why not just use cluster_resource_map_[local_node_id]?

I think cluster_resource_map_ is a simple map of resource type and quantities used for making scheduling decisions, while local_available_resources_ handles the resource id allocation and releases at the local raylet and is used for resource id bookkeeping. Maybe we could store resource ids in cluster_resource_map_ and use a single structure for both scheduling and bookkeeping.. @robertnishihara would be able to provide better insight.

If we can remove local_available_resources_, I beleive things will get more simple and clearer.

[robertnishihara]

@robertnishihara do you know why we need this local_available_resources_ while we already have cluster_resource_map_? why not just use cluster_resource_map_[local_node_id]?

@raulchen, the reason is that we do fine-grained bookkeeping for the local resources (we keep track of resource IDs), whereas for cluster resources, we only keep track of the aggregate resource quantities. There's probably a better way to do it though.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15724/
Test PASSed.

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/15723/
Test PASSed.