Skip to content

For strategies.yaml, add peering_ring with go_direct, and Au tests for peering_ring. #7925

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ywkaras merged 1 commit into from
Sep 8, 2021
Merged

For strategies.yaml, add peering_ring with go_direct, and Au tests for peering_ring. #7925

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 11 additions & 2 deletions doc/admin-guide/files/strategies.yaml.en.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -190,12 +190,21 @@ Each **strategy** in the list may using the following parameters::

#. **exhaust_ring**: when a host normally selected by the policy fails, another host is selected from the same group. A new group is not selected until all hosts on the previous group have been exhausted
#. **alternate_ring**: retry hosts are selected from groups in an alternating group fashion.
#. **peering_ring** This mode is only implemented for a policy of **consistent_hash** and requires that two host groups are defined. The first host group is a list of peer caches and "this" host itself, the second group is a list of upstream caches. Parents are always selected from the peer list however, if the selected parent is "this" host itself a new parent from the upstream list is chosen. In addition, if any peer host is unreachable or times out, a host from the upstream list is chosen for retries.
#. **peering_ring**: This mode is only implemented for a policy of **consistent_hash** and requires that one or two
host groups are defined. The first host group is a list of peer caches and "this" host itself, the (optional) second
group is a list of upstream caches. Parents are always selected from the peer list however, if the selected parent is
"this" host itself a new parent from the upstream list is chosen. If the second group is omitted, and **go_direct**
is **true**, the upstream "list" has one element,
the host in the remapped URL. In addition, if any peer host is unreachable or times out, a host from the upstream
list is chosen for retries. Because the peer hosts may at times not have consistent up/down markings for the other
peers, requests may be looped sometimes. So it's best to use :ts:cv:`proxy.config.http.insert_request_via_str` and :ts:cv:`proxy.config.http.max_proxy_cycles` to stop looping.

- **response_codes**: Part of the **failover** map. This is a list of **http** response codes that may be used for **simple retry**.
- **markdown_codes**: Part of the **failover** map. This is a list of **http** response codes that may be used for **unavailable retry** which will cause a parent markdown.
- **health_check**: Part of the **failover** map. A list of health checks. **passive** is the default and means that the state machine marks down **hosts** when a transaction timeout or connection error is detected. **passive** is always used by the next hop strategies. **active** means that some external process may actively health check the hosts using the defined **health check url** and mark them down using **traffic_ctl**.

- **self**: Part of the **failover** map. This can only be used when **ring_mode** is **peering_ring**. This is the hostname of the host in the (first) group of peers that is the local host |TS| runs on.
(**self** should only be necessary when the local hostname can only be translated to an IP address
with a DNS lookup.)

Example:
::
Expand Down
231 changes: 123 additions & 108 deletions proxy/http/remap/NextHopConsistentHash.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,8 @@ NextHopConsistentHash::getHashKey(uint64_t sm_id, HttpRequestData *hrdata, ATSHa
void
NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
{
uint32_t const NO_RING_USE_POST_REMAP = uint32_t(0) - 1;

HttpSM *sm = reinterpret_cast<HttpSM *>(txnp);
ParentResult *result = &sm->t_state.parent_result;
HttpRequestData request_info = sm->t_state.request_data;
Expand All @@ -232,7 +234,6 @@ NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
TSHostStatus host_stat = TSHostStatus::TS_HOST_STATUS_INIT;
HostStatRec *hst = nullptr;
Machine *machine = Machine::instance();
;

if (result->line_number == -1 && result->result == PARENT_UNDEFINED) {
firstcall = true;
Expand All @@ -259,10 +260,14 @@ NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
}
break;
case NH_PEERING_RING:
ink_assert(groups == 2);
// look for the next parent on the
// upstream ring.
result->last_group = cur_ring = 1;
if (groups == 1) {
result->last_group = cur_ring = NO_RING_USE_POST_REMAP;
} else {
ink_assert(groups == 2);
// look for the next parent on the
// upstream ring.
result->last_group = cur_ring = 1;
}
break;
case NH_EXHAUST_RING:
default:
Expand All @@ -275,125 +280,135 @@ NextHopConsistentHash::findNextHop(TSHttpTxn txnp, void *ih, time_t now)
}
}

// Do the initial parent look-up.
hash_key = getHashKey(sm_id, &request_info, &hash);

do { // search until we've selected a different parent if !firstcall
std::shared_ptr<ATSConsistentHash> r = rings[cur_ring];
hostRec = chash_lookup(r, hash_key, &result->chashIter[cur_ring], &wrapped, &hash, &result->chash_init[cur_ring],
&result->mapWrapped[cur_ring], sm_id);
wrap_around[cur_ring] = wrapped;
lookups++;
// the 'available' flag is maintained in 'host_groups' and not the hash ring.
if (hostRec) {
pRec = host_groups[hostRec->group_index][hostRec->host_index];
if (firstcall) {
hst = (pRec) ? pStatus.getHostStatus(pRec->hostname.c_str()) : nullptr;
result->first_choice_status = (hst) ? hst->status : TSHostStatus::TS_HOST_STATUS_UP;
// if peering and the selected host is myself, change rings and search for an upstream
// parent.
if (ring_mode == NH_PEERING_RING && machine->is_self(pRec->hostname)) {
// switch to the upstream ring.
cur_ring = 1;
continue;
}
break;
}
} else {
pRec = nullptr;
}
} while (pRec && result->hostname && strcmp(pRec->hostname.c_str(), result->hostname) == 0);

NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] Initial parent lookups: %d", sm_id, lookups);
if (cur_ring != NO_RING_USE_POST_REMAP) {
// Do the initial parent look-up.
hash_key = getHashKey(sm_id, &request_info, &hash);

// ----------------------------------------------------------------------------------------------------
// Validate initial parent look-up and perform additional look-ups if required.
// ----------------------------------------------------------------------------------------------------

hst = (pRec) ? pStatus.getHostStatus(pRec->hostname.c_str()) : nullptr;
host_stat = (hst) ? hst->status : TSHostStatus::TS_HOST_STATUS_UP;
// if the config ignore_self_detect is set to true and the host is down due to SELF_DETECT reason
// ignore the down status and mark it as available
if ((pRec && ignore_self_detect) && (hst && hst->status == TS_HOST_STATUS_DOWN)) {
if (hst->reasons == Reason::SELF_DETECT) {
host_stat = TS_HOST_STATUS_UP;
}
}
if (!pRec || (pRec && !pRec->available.load()) || host_stat == TS_HOST_STATUS_DOWN) {
do {
// check if an unavailable server is now retryable, use it.
if (pRec && !pRec->available.load() && host_stat == TS_HOST_STATUS_UP) {
// check if the host is retryable. It's retryable if the retry window has elapsed
_now == 0 ? _now = time(nullptr) : _now = now;
if ((pRec->failedAt.load() + retry_time) < static_cast<unsigned>(_now)) {
NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] next hop %s, retriers: %d", sm_id, pRec->hostname.c_str(), pRec->retriers());
if (pRec->retriers.inc(max_retriers)) {
nextHopRetry = true;
result->last_parent = pRec->host_index;
result->last_lookup = pRec->group_index;
result->retry = nextHopRetry;
result->result = PARENT_SPECIFIED;
NH_Debug(NH_DEBUG_TAG,
"[%" PRIu64 "] next hop %s is now retryable, marked it available, retriers: %d, max_retriers: %d.", sm_id,
pRec->hostname.c_str(), pRec->retriers(), max_retriers);
break;
}
}
}
switch (ring_mode) {
case NH_ALTERNATE_RING:
if (pRec && groups > 0) {
cur_ring = (pRec->group_index + 1) % groups;
} else {
cur_ring = 0;
}
break;
case NH_EXHAUST_RING:
default:
if (wrap_around[cur_ring] && groups > 1) {
cur_ring = (cur_ring + 1) % groups;
}
break;
}
do { // search until we've selected a different parent if !firstcall
std::shared_ptr<ATSConsistentHash> r = rings[cur_ring];
hostRec = chash_lookup(r, hash_key, &result->chashIter[cur_ring], &wrapped, &hash, &result->chash_init[cur_ring],
&result->mapWrapped[cur_ring], sm_id);
wrap_around[cur_ring] = wrapped;
lookups++;
// the 'available' flag is maintained in 'host_groups' and not the hash ring.
if (hostRec) {
pRec = host_groups[hostRec->group_index][hostRec->host_index];
hst = (pRec) ? pStatus.getHostStatus(pRec->hostname.c_str()) : nullptr;
host_stat = (hst) ? hst->status : TSHostStatus::TS_HOST_STATUS_UP;
// if the config ignore_self_detect is set to true and the host is down due to SELF_DETECT reason
// ignore the down status and mark it as available
if ((pRec && ignore_self_detect) && (hst && hst->status == TS_HOST_STATUS_DOWN)) {
if (hst->reasons == Reason::SELF_DETECT) {
host_stat = TS_HOST_STATUS_UP;
pRec = host_groups[hostRec->group_index][hostRec->host_index];
if (firstcall) {
hst = (pRec) ? pStatus.getHostStatus(pRec->hostname.c_str()) : nullptr;
result->first_choice_status = (hst) ? hst->status : TSHostStatus::TS_HOST_STATUS_UP;
// if peering and the selected host is myself, change rings and search for an upstream
// parent.
if (ring_mode == NH_PEERING_RING && (pRec->self || machine->is_self(pRec->hostname.c_str()))) {
if (groups == 1) {
// use host from post-remap URL
cur_ring = NO_RING_USE_POST_REMAP;
pRec = nullptr;
} else {
// switch to the upstream ring.
cur_ring = 1;
continue;
}
}
}
NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] Selected a new parent: %s, available: %s, wrapped: %s, lookups: %d.", sm_id,
pRec->hostname.c_str(), (pRec->available.load()) ? "true" : "false", (wrapped) ? "true" : "false", lookups);
// use available host.
if (pRec->available.load() && host_stat == TS_HOST_STATUS_UP) {
break;
}
} else {
pRec = nullptr;
}
bool all_wrapped = true;
for (uint32_t c = 0; c < groups; c++) {
if (wrap_around[c] == false) {
all_wrapped = false;
} while (pRec && result->hostname && strcmp(pRec->hostname.c_str(), result->hostname) == 0);

NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] Initial parent lookups: %d", sm_id, lookups);

// ----------------------------------------------------------------------------------------------------
// Validate initial parent look-up and perform additional look-ups if required.
// ----------------------------------------------------------------------------------------------------

if (cur_ring != NO_RING_USE_POST_REMAP) {
hst = (pRec) ? pStatus.getHostStatus(pRec->hostname.c_str()) : nullptr;
host_stat = (hst) ? hst->status : TSHostStatus::TS_HOST_STATUS_UP;
// if the config ignore_self_detect is set to true and the host is down due to SELF_DETECT reason
// ignore the down status and mark it as available
if ((pRec && ignore_self_detect) && (hst && hst->status == TS_HOST_STATUS_DOWN)) {
if (hst->reasons == Reason::SELF_DETECT) {
host_stat = TS_HOST_STATUS_UP;
}
}
if (all_wrapped) {
NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] No available parents.", sm_id);
if (pRec) {
pRec = nullptr;
}
break;
if (!pRec || (pRec && !pRec->available.load()) || host_stat == TS_HOST_STATUS_DOWN) {
do {
// check if an unavailable server is now retryable, use it.
if (pRec && !pRec->available.load() && host_stat == TS_HOST_STATUS_UP) {
// check if the host is retryable. It's retryable if the retry window has elapsed
_now == 0 ? _now = time(nullptr) : _now = now;
if ((pRec->failedAt.load() + retry_time) < static_cast<unsigned>(_now)) {
NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] next hop %s, retriers: %d", sm_id, pRec->hostname.c_str(), pRec->retriers());
if (pRec->retriers.inc(max_retriers)) {
nextHopRetry = true;
result->last_parent = pRec->host_index;
result->last_lookup = pRec->group_index;
result->retry = nextHopRetry;
result->result = PARENT_SPECIFIED;
NH_Debug(NH_DEBUG_TAG,
"[%" PRIu64 "] next hop %s is now retryable, marked it available, retriers: %d, max_retriers: %d.", sm_id,
pRec->hostname.c_str(), pRec->retriers(), max_retriers);
break;
}
}
}
switch (ring_mode) {
case NH_ALTERNATE_RING:
if (pRec && groups > 0) {
cur_ring = (pRec->group_index + 1) % groups;
} else {
cur_ring = 0;
}
break;
case NH_EXHAUST_RING:
default:
if (wrap_around[cur_ring] && groups > 1) {
cur_ring = (cur_ring + 1) % groups;
}
break;
}
std::shared_ptr<ATSConsistentHash> r = rings[cur_ring];
hostRec = chash_lookup(r, hash_key, &result->chashIter[cur_ring], &wrapped, &hash, &result->chash_init[cur_ring],
&result->mapWrapped[cur_ring], sm_id);
wrap_around[cur_ring] = wrapped;
lookups++;
if (hostRec) {
pRec = host_groups[hostRec->group_index][hostRec->host_index];
hst = (pRec) ? pStatus.getHostStatus(pRec->hostname.c_str()) : nullptr;
host_stat = (hst) ? hst->status : TSHostStatus::TS_HOST_STATUS_UP;
// if the config ignore_self_detect is set to true and the host is down due to SELF_DETECT reason
// ignore the down status and mark it as available
if ((pRec && ignore_self_detect) && (hst && hst->status == TS_HOST_STATUS_DOWN)) {
if (hst->reasons == Reason::SELF_DETECT) {
host_stat = TS_HOST_STATUS_UP;
}
}
NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] Selected a new parent: %s, available: %s, wrapped: %s, lookups: %d.", sm_id,
pRec->hostname.c_str(), (pRec->available.load()) ? "true" : "false", (wrapped) ? "true" : "false", lookups);
// use available host.
if (pRec->available.load() && host_stat == TS_HOST_STATUS_UP) {
break;
}
} else {
pRec = nullptr;
}
bool all_wrapped = true;
for (uint32_t c = 0; c < groups; c++) {
if (wrap_around[c] == false) {
all_wrapped = false;
}
}
if (all_wrapped) {
NH_Debug(NH_DEBUG_TAG, "[%" PRIu64 "] No available parents.", sm_id);
if (pRec) {
pRec = nullptr;
}
break;
}
} while (!pRec || (pRec && !pRec->available.load()) || host_stat == TS_HOST_STATUS_DOWN);
}
} while (!pRec || (pRec && !pRec->available.load()) || host_stat == TS_HOST_STATUS_DOWN);
}
}

// ----------------------------------------------------------------------------------------------------
Expand Down
31 changes: 27 additions & 4 deletions proxy/http/remap/NextHopSelectionStrategy.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ bool
NextHopSelectionStrategy::Init(ts::Yaml::Map &n)
{
NH_Debug(NH_DEBUG_TAG, "calling Init()");
std::string self_host;
Copy link
Contributor

@jrushford jrushford Jul 14, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ywkaras Consider adding this new field to the unit testing, see proxy/remap/http/unit-tests/test_NextHopConsistentHash.cc and peering.yaml in the same unit-tests directory.

bool self_host_used = false;

try {
if (n["scheme"]) {
Expand Down Expand Up @@ -108,7 +110,12 @@ NextHopSelectionStrategy::Init(ts::Yaml::Map &n)
} else if (ring_mode_val == exhaust_rings) {
ring_mode = NH_EXHAUST_RING;
} else if (ring_mode_val == peering_rings) {
ring_mode = NH_PEERING_RING;
ring_mode = NH_PEERING_RING;
YAML::Node self_node = failover_node["self"];
if (self_node) {
Copy link
Contributor

@jrushford jrushford Aug 16, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ywkaras Is this correct, self is part of the failover node`. I'm not sure how this works in identifying the self host. Maybe I'm missing something. In testing, I'm looping when the self host is selected.
I used 'self: my_self_hostname.cdn.com'. This doesn't parse, no error is logged but the strategy is destroyed in the NextHopStrategyFactory:143

self_host = self_node.Scalar();
NH_Debug(NH_DEBUG_TAG, "%s is self", self_host.c_str());
}
} else {
ring_mode = NH_ALTERNATE_RING;
NH_Note("Invalid 'ring_mode' value, '%s', for the strategy named '%s', using default '%s'.", ring_mode_val.c_str(),
Expand Down Expand Up @@ -214,8 +221,14 @@ NextHopSelectionStrategy::Init(ts::Yaml::Map &n)
std::shared_ptr<HostRecord> host_rec = std::make_shared<HostRecord>(hosts_list[hst].as<HostRecord>());
host_rec->group_index = grp;
host_rec->host_index = hst;
if (mach->is_self(host_rec->hostname)) {
if ((self_host == host_rec->hostname) || mach->is_self(host_rec->hostname.c_str())) {
if (ring_mode == NH_PEERING_RING && grp != 0) {
throw std::invalid_argument("self host (" + self_host +
") can only appear in first host group for peering ring mode");
}
h_stat.setHostStatus(host_rec->hostname.c_str(), TSHostStatus::TS_HOST_STATUS_DOWN, 0, Reason::SELF_DETECT);
Copy link
Contributor

@jrushford jrushford Aug 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ywkaras the self detection doesn't seem to work now. Host status no-longer has a SELF_DETECT down for the self_host. I'm using the peering ring mode.

host_rec->self = true;
self_host_used = true;
}
hosts_inner.push_back(std::move(host_rec));
num_parents++;
Expand All @@ -226,14 +239,24 @@ NextHopSelectionStrategy::Init(ts::Yaml::Map &n)
}
}
}
if (!self_host.empty() && !self_host_used) {
throw std::invalid_argument("self host (" + self_host + ") does not appear in the first (peer) group");
}
} catch (std::exception &ex) {
NH_Error("Error parsing the strategy named '%s' due to '%s', this strategy will be ignored.", strategy_name.c_str(), ex.what());
return false;
}

if (ring_mode == NH_PEERING_RING) {
if (groups != 2) {
NH_Error("ring mode is '%s', requires two host groups (peering group and an upstream group).", peering_rings.data());
if (groups == 1) {
if (!go_direct) {
NH_Error("when ring mode is '%s', go_direct must be true when there is only one host group.", peering_rings.data());
return false;
}
} else if (groups != 2) {
NH_Error("when ring mode is '%s', requires two host groups (peering group and an upstream group),"
" or just a single peering group with go_direct.",
peering_rings.data());
return false;
}
if (policy_type != NH_CONSISTENT_HASH) {
Expand Down
1 change: 1 addition & 0 deletions proxy/http/remap/NextHopSelectionStrategy.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ struct HostRecord : ATSConsistentHashNode {
std::string hash_string;
int host_index;
int group_index;
bool self = false;
std::vector<std::shared_ptr<NHProtocol>> protocols;
pRetriers retriers;

Expand Down
6 changes: 4 additions & 2 deletions proxy/http/remap/unit-tests/peering.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,10 +96,10 @@ groups:
weight: 0.25
- <<: *p2
weight: 0.25
- <<: *p2
weight: 0.25
- <<: *p3
weight: 0.25
- <<: *p4
weight: 0.25
- &g2
- <<: *m1
weight: 0.33
Expand All @@ -121,6 +121,8 @@ strategies:
max_simple_retries: 2
ring_mode:
peering_ring
self:
p3.bar.com
response_codes: # defines the responses codes for failover in exhaust_ring mode
- 404
health_check: # specifies the list of healthchecks that should be considered for failover. A list of enums: 'passive' or 'active'
Expand Down
Loading