Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GCS]Add gcs resource scheduler #13072

Merged
merged 32 commits into from
Jan 14, 2021
Merged

[GCS]Add gcs resource scheduler #13072

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
bd47564
add part code
Dec 10, 2020
b78c894
rebase master
Dec 12, 2020
4170043
add part code
Dec 10, 2020
8578907
add part code
Dec 11, 2020
58b5d85
fix code style
Dec 13, 2020
4897d7f
fix ut bug
Dec 13, 2020
c6333ce
add part code
Dec 9, 2020
600bd43
add part code
Dec 10, 2020
3a788e6
add part code
Dec 10, 2020
b07f369
add part code
Dec 14, 2020
003d6e2
add part code
Dec 14, 2020
d68e9df
rebase master
Dec 24, 2020
6cec27c
add part code
Dec 27, 2020
910d186
rebase master
Dec 27, 2020
b78a1d2
add part code
Dec 27, 2020
d5bc7ff
add part code
Dec 28, 2020
7dd33b9
add part code
Dec 28, 2020
71a8547
add part code
Dec 29, 2020
fcf4bab
add part code
Dec 29, 2020
1a003bc
add part code
Dec 30, 2020
368dab5
add part code
Dec 30, 2020
c13ebfe
add part code
Dec 30, 2020
6d77661
add part code
Dec 30, 2020
ff5bf3b
add part code
Dec 30, 2020
0cab178
fix review comments
Jan 5, 2021
9101161
fix review comments
Jan 7, 2021
3ce8e41
fix review comment
Jan 7, 2021
4d1bc9b
fix review comments
Jan 7, 2021
ec7667e
fix review comment
Jan 7, 2021
351d4ec
add testcase
Jan 7, 2021
d2f9c85
fix review comments
Jan 7, 2021
f58c95d
fix review comment
Jan 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
add part code
  • Loading branch information
灵洵 committed Dec 29, 2020
commit fcf4bab98737d93c0046fefa84586f90e37ce60e
13 changes: 5 additions & 8 deletions src/ray/gcs/gcs_server/gcs_placement_group_scheduler.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,7 @@ ScheduleMap GcsStrictPackStrategy::Schedule(
const std::unique_ptr<ScheduleContext> &context) {
const auto &required_resources = GetRequiredResourcesFromBundles(bundles);
const auto &selected_nodes = context->gcs_resource_scheduler_.Schedule(
required_resources, SchedulingPolicy(SchedulingType::STRICT_PACK),
[](const NodeID &) { return true; });
required_resources, SchedulingType::STRICT_PACK);
return GenerateScheduleMap(bundles, selected_nodes);
}

Expand All @@ -78,9 +77,8 @@ ScheduleMap GcsPackStrategy::Schedule(
// First fill up a node. If the node resource is insufficient, select a new node.
// TODO(ffbin): We will speed this up in next PR. Currently it is a double for loop.
const auto &required_resources = GetRequiredResourcesFromBundles(bundles);
const auto &selected_nodes = context->gcs_resource_scheduler_.Schedule(
required_resources, SchedulingPolicy(SchedulingType::PACK),
[](const NodeID &) { return true; });
const auto &selected_nodes =
context->gcs_resource_scheduler_.Schedule(required_resources, SchedulingType::PACK);
return GenerateScheduleMap(bundles, selected_nodes);
}

Expand All @@ -89,8 +87,7 @@ ScheduleMap GcsSpreadStrategy::Schedule(
const std::unique_ptr<ScheduleContext> &context) {
const auto &required_resources = GetRequiredResourcesFromBundles(bundles);
const auto &selected_nodes = context->gcs_resource_scheduler_.Schedule(
required_resources, SchedulingPolicy(SchedulingType::SPREAD),
[](const NodeID &) { return true; });
required_resources, SchedulingType::SPREAD);
return GenerateScheduleMap(bundles, selected_nodes);
}

Expand All @@ -112,7 +109,7 @@ ScheduleMap GcsStrictSpreadStrategy::Schedule(

const auto &required_resources = GetRequiredResourcesFromBundles(bundles);
const auto &selected_nodes = context->gcs_resource_scheduler_.Schedule(
required_resources, SchedulingPolicy(SchedulingType::STRICT_SPREAD),
required_resources, SchedulingType::STRICT_SPREAD,
[nodes_in_use](const NodeID &node_id) { return nodes_in_use.count(node_id) == 0; });
ffbin marked this conversation as resolved.
Show resolved Hide resolved
return GenerateScheduleMap(bundles, selected_nodes);
}
Expand Down
13 changes: 5 additions & 8 deletions src/ray/gcs/gcs_server/gcs_resource_scheduler.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,6 @@ double LeastResourceScorer::MakeGrade(const ResourceSet &required_resources,
if (!required_resources.GetResourceAmountMap().empty()) {
node_score /= required_resources.GetResourceAmountMap().size();
ffbin marked this conversation as resolved.
Show resolved Hide resolved
}

RAY_LOG(INFO) << "LeastResourceScorer::MakeGrade, available_resources is "
<< available_resources.ToString() << ", required_resources is "
<< required_resources.ToString() << ", node_score is " << node_score;
return node_score;
}

Expand All @@ -59,7 +55,8 @@ double LeastResourceScorer::Calculate(const FractionalResourceQuantity &requeste

/////////////////////////////// Begin of GcsResourceScheduler ///////////////////////////
std::vector<NodeID> GcsResourceScheduler::Schedule(
const std::vector<ResourceSet> &required_resources, const SchedulingPolicy &policy,
const std::vector<ResourceSet> &required_resources,
const SchedulingType &scheduling_type,
const std::function<bool(const NodeID &)> &node_filter_func) {
const auto &cluster_resources = gcs_resource_manager_.GetClusterResources();

Expand All @@ -77,7 +74,7 @@ std::vector<NodeID> GcsResourceScheduler::Schedule(

// Score and rank nodes.
std::vector<NodeID> result;
switch (policy.type_) {
switch (scheduling_type) {
case SPREAD:
SpreadSchedule(to_schedule_resources, candidate_nodes, &result);
break;
Expand All @@ -91,7 +88,7 @@ std::vector<NodeID> GcsResourceScheduler::Schedule(
StrictPackSchedule(to_schedule_resources, candidate_nodes, &result);
break;
default:
RAY_LOG(FATAL) << "Unsupported policy type: " << policy.type_;
RAY_LOG(FATAL) << "Unsupported scheduling type: " << scheduling_type;
break;
}
return result;
Expand All @@ -103,7 +100,7 @@ absl::flat_hash_set<NodeID> GcsResourceScheduler::FilterCandidateNodes(
absl::flat_hash_set<NodeID> result;
for (const auto &iter : cluster_resources) {
const auto &node_id = iter.first;
if (node_filter_func(node_id)) {
if (node_filter_func == nullptr || node_filter_func(node_id)) {
result.emplace(node_id);
}
}
Expand Down
26 changes: 3 additions & 23 deletions src/ray/gcs/gcs_server/gcs_resource_scheduler.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,6 @@ enum SchedulingType {

typedef std::pair<NodeID, double> NodeScore;

class SchedulingPolicy {
public:
SchedulingPolicy(const SchedulingType &type) : type_(type) {}

const SchedulingType type_;
};

/// NodeScorer is a scorer to make a grade to the node, which is used for scheduling
/// decision.
class NodeScorer {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not clear to me, how this works out with gcs_actor_schedule_strategy.h ?
https://github.com/ray-project/ray/pull/13156/files#diff-843c9a23aa73419d34f34562947608459816123e02d1f48a7171e897e115c0ff

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gcs_actor_schedule_strategy.h will use gcs_resource_scheduler.h to schedule actor.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel having NodeScorer & schedule_strategy in different files would cause split brain problem. NodeScorer is also affecting scheduling strategy?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NodeScorer & schedule_strategy are decoupled.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, on a second look, this makes sense

Expand Down Expand Up @@ -75,20 +68,6 @@ class LeastResourceScorer : public NodeScorer {
const FractionalResourceQuantity &available);
};

// class ResourceScheduleContext {
// public:
// virtual ~ResourceScheduleContext() {}
//
// virtual bool IsNodeSelected(const NodeID &node_id) {
// return std::find(selected_nodes.begin(), selected_nodes.end(), node_id) !=
// selected_nodes.end();
// }
//
// virtual bool IsNodeSchedulable(const NodeID &node_id) { return true; }
//
// std::vector<NodeID> selected_nodes;
//};

/// Gcs resource scheduler implementation.
/// Non-thread safe.
class GcsResourceScheduler {
Expand All @@ -100,8 +79,9 @@ class GcsResourceScheduler {
virtual ~GcsResourceScheduler() = default;

std::vector<NodeID> Schedule(
const std::vector<ResourceSet> &required_resources, const SchedulingPolicy &policy,
const std::function<bool(const NodeID &)> &node_filter_func);
const std::vector<ResourceSet> &required_resources,
const SchedulingType &scheduling_type,
const std::function<bool(const NodeID &)> &node_filter_func = nullptr);

private:
absl::flat_hash_set<NodeID> FilterCandidateNodes(
Expand Down