Support set local region for RegionAwareEnsemblePlacementPolicy #3248

wangjialing218 · 2022-04-26T07:34:38Z

Motivation

When RegionAwareEnsemblePlacementPolicy is enabled, write set of one entry contains bookie nodes in different region.
When bookkeeper client read entries, the policy provide a read sequence to read from bookies in the same region first. see

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Line 834 in 4debbbf

DistributionSchedule.WriteSet reorderReadSequenceWithRegion(

It depend on myRegion which indicate the local region of bookkeeper client (mostly pulsar broker).
Local region is not a configurable value, and it's generated by local host address. see

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Line 208 in 4debbbf

bn = createDummyLocalBookieNode(InetAddress.getLocalHost().getHostAddress());

It's okey when there is a bookie in the same host of broker.
But in most case, brokers were deployed in other nodes (or pod in kubernetes), and bookkeeper client will fail to get local region value.

Changes

Add configuration of local region for bookkeeper client.

hangc0276

For your motivation, If we read all the entries from the configured region, it may cause hot bookie, and increase the quarantine rate of the hot bookie.

hangc0276 · 2022-06-24T07:17:29Z

...erver/src/test/java/org/apache/bookkeeper/client/TestRegionAwareEnsemblePlacementPolicy.java

        basicReorderReadSequenceWithLocalRegionTest("region2", false);
    }

    @Test
    public void testBasicReorderReadLACSequenceWithLocalRegion() throws Exception {
+        prepareNetworkTopologyForReorderTests("region2");


We should both test read and write.

For your motivation, If we read all the entries from the configured region, it may cause hot bookie, and increase the quarantine rate of the hot bookie.

As current policy, read sequence is as follows

bookkeeper/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/RackawareEnsemblePlacementPolicyImpl.java

Line 810 in 4debbbf

* 1. available (local) bookies

This motivation just add a configuration to set the local region as myRegion for client, bookies in the same region will be orderd as 1. available (local) bookies, when these bookies become unavailable or slow, they will be orderd as 3. R* remaining (local) bookies or 6. slow bookies.

We should both test read and write.

This motivation just add a configuration related to the read order of bookies in a writeSet. IMO we do not need test write.

StevenLuMT · 2022-07-26T13:17:10Z

rerun failure checks

StevenLuMT · 2022-07-26T16:21:44Z

...er-server/src/main/java/org/apache/bookkeeper/client/RegionAwareEnsemblePlacementPolicy.java

@@ -173,6 +174,10 @@ public RegionAwareEnsemblePlacementPolicy initialize(ClientConfiguration conf,
        super.initialize(conf, optionalDnsResolver, timer, featureProvider, statsLogger, bookieAddressResolver)
                .withDefaultRack(NetworkTopology.DEFAULT_REGION_AND_RACK);
        myRegion = getLocalRegion(localNode);
+        String localRegion = conf.getString(REPP_LOCAL_REGION, null);


simple code
String localRegion = conf.getString(REPP_LOCAL_REGION);
instead of
String localRegion = conf.getString(REPP_LOCAL_REGION, null);

I'm not sure whether throwExceptionOnMissing could be set true in some condition.

StevenLuMT · 2022-07-26T16:22:30Z

...er-server/src/main/java/org/apache/bookkeeper/client/RegionAwareEnsemblePlacementPolicy.java

@@ -79,7 +80,7 @@ public class RegionAwareEnsemblePlacementPolicy extends RackawareEnsemblePlaceme
    protected Feature disableDurabilityFeature;
    private int lastRegionIndex = 0;

-    RegionAwareEnsemblePlacementPolicy() {
+    protected RegionAwareEnsemblePlacementPolicy() {


why not change it to public?

Change to protected so that we could extends RegionAwareEnsemblePlacementPolicy and implement some addtion policy in broker side in future.
IMO no need change to public.

StevenLuMT · 2022-08-24T08:02:25Z

fix old workflow,please see #3455 for detail

dlg99 · 2022-12-07T18:02:58Z

What alternative solutions have you looked at?

It's okey when there is a bookie in the same host of broker.

AFACT, the dummy bookie node is created to make use of the resolver API that takes BookieNode as a parameter.
After that it goes through the same resolution mechanism as any bookie; so if you use script-based mapping you can update it to resolve the client's address to region/rack etc, it does not really matter if the client is co-hosted with a bookie.

wangjialing218 · 2023-01-11T02:57:28Z

What alternative solutions have you looked at?

In synchronous geo-replication solution, suppose we have 4 bookies in two regions, bk1 and bk2 with rack setting /region1/rack, bk3 and bk4 with rack setting /region2/rack。
A broker in region1 need to read ledger data and the writeSet is {bk1,bk3}, it's better to let broker read from bk1 which is in same region to avoid across region tracffic. If the topic unloaded to another broker in region2, the broker could read ledger data from bk3 instead.

AFACT, the dummy bookie node is created to make use of the resolver API that takes BookieNode as a parameter. After that it goes through the same resolution mechanism as any bookie; so if you use script-based mapping you can update it to resolve the client's address to region/rack etc, it does not really matter if the client is co-hosted with a bookie.

It's okey to resolve the broker's address to region/rack to achieve, we need to configure the resolver and it's not an easy way if brokers deployed in kubernetes.
This PR provide a bookkeeper client configuration reppLocalRegion for broker, so we could add region-aware setting for each broker according to where the broker is placement by command or configuration.

hangc0276 · 2023-08-24T09:34:50Z

It's okey to resolve the broker's address to region/rack to achieve, we need to configure the resolver and it's not an easy way if brokers deployed in kubernetes.

@wangjialing218 It is easy to configure by bin/pulsar-admin bookies set-bookie-rack command

If we introduce a configuration reppLocalRegion for the broker, and the broker is scheduled to another region by k8s, we still need to change the reppLocalRegion in conf/broker.conf. It will be hard to maintain.

hangc0276 requested changes Jun 24, 2022

View reviewed changes

hangc0276 requested review from dlg99, eolivelli, shoothzj and zymap July 25, 2022 01:35

hangc0276 assigned wangjialing218 Jul 25, 2022

hangc0276 added area/client type/improvement labels Jul 25, 2022

hangc0276 added this to the 4.16.0 milestone Jul 25, 2022

StevenLuMT reviewed Jul 26, 2022

View reviewed changes

wangjialing added 2 commits July 29, 2022 15:57

support set local region for RegionAwareEnsemblePlacementPolicy

6dff264

fix test code

d4fd1ce

wangjialing218 force-pushed the branche-wjl branch from 5c75626 to d4fd1ce Compare July 29, 2022 08:00

hangc0276 modified the milestones: 4.16.0, 4.17.0 Jul 31, 2022

wangjialing218 closed this Nov 4, 2022

wangjialing218 reopened this Nov 4, 2022

hangc0276 modified the milestones: 4.17.0, 4.16.0 Jan 11, 2023

hangc0276 modified the milestones: 4.16.0, 4.17.0 Mar 13, 2023

eolivelli removed this from the 4.17.0 milestone May 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support set local region for RegionAwareEnsemblePlacementPolicy #3248

Support set local region for RegionAwareEnsemblePlacementPolicy #3248

wangjialing218 commented Apr 26, 2022

hangc0276 left a comment

hangc0276 Jun 24, 2022

wangjialing218 Jul 26, 2022 •

edited

Loading

StevenLuMT commented Jul 26, 2022

StevenLuMT Jul 26, 2022

wangjialing218 Jul 27, 2022

StevenLuMT Jul 26, 2022

wangjialing218 Jul 27, 2022

StevenLuMT commented Aug 24, 2022

dlg99 commented Dec 7, 2022

wangjialing218 commented Jan 11, 2023

hangc0276 commented Aug 24, 2023

Support set local region for RegionAwareEnsemblePlacementPolicy #3248

Are you sure you want to change the base?

Support set local region for RegionAwareEnsemblePlacementPolicy #3248

Conversation

wangjialing218 commented Apr 26, 2022

Motivation

Changes

hangc0276 left a comment

Choose a reason for hiding this comment

hangc0276 Jun 24, 2022

Choose a reason for hiding this comment

wangjialing218 Jul 26, 2022 • edited Loading

Choose a reason for hiding this comment

StevenLuMT commented Jul 26, 2022

StevenLuMT Jul 26, 2022

Choose a reason for hiding this comment

wangjialing218 Jul 27, 2022

Choose a reason for hiding this comment

StevenLuMT Jul 26, 2022

Choose a reason for hiding this comment

wangjialing218 Jul 27, 2022

Choose a reason for hiding this comment

StevenLuMT commented Aug 24, 2022

dlg99 commented Dec 7, 2022

wangjialing218 commented Jan 11, 2023

hangc0276 commented Aug 24, 2023

wangjialing218 Jul 26, 2022 •

edited

Loading