Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix][meta] Fix ephemeral handling of ZK nodes and fix MockZooKeeper ephemeral and ZK stat handling #23988

Merged
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
ee22969
[fix][metadata] Fix ephemeral status of ZK nodes
lhotari Feb 14, 2025
c45c6d6
Run pulsar-metadata tests with MockZooKeeper
lhotari Feb 14, 2025
cd5610a
Improve test names
lhotari Feb 14, 2025
244a65d
Fix Stat handling in MockZooKeeper
lhotari Feb 14, 2025
d9ec2d1
Cleanup watchers tied to a particular ZK session in MockZooKeeper
lhotari Feb 14, 2025
153289e
Improve MockZNode handling
lhotari Feb 14, 2025
69ca2df
Pass the overridden session id to the executor thread
lhotari Feb 14, 2025
d60b9bc
Don't use direct executor with MockZooKeeper since it could cause ord…
lhotari Feb 14, 2025
2298de6
Ensure that connection strings don't change during a single execution
lhotari Feb 14, 2025
0f5c46f
Improve checkContainers so that it's only called for ZK
lhotari Feb 14, 2025
701dfad
Skip testDeleteUnusedDirectories for MockZooKeeper
lhotari Feb 14, 2025
13ac174
Improve assertions in getChildren test
lhotari Feb 14, 2025
368993b
Fix MockZooKeeper's getChildren implementation
lhotari Feb 14, 2025
ea760b4
Consistently unlock before calling callbacks
lhotari Feb 14, 2025
4989e21
Restore separate mockZooKeeperGlobal instance
lhotari Feb 14, 2025
9cbd80f
Show test name for BrokerServiceLookupTest in IntelliJ
lhotari Feb 14, 2025
ee670b3
Optimize MockZooKeeper getChildren
lhotari Feb 15, 2025
53e587e
Improve BaseMetadataStoreTest: reduce duplication
lhotari Feb 15, 2025
aee6696
Use zkImpl so that provider name shows up in IntelliJ test runner
lhotari Feb 15, 2025
38f4b77
Also test MockZooKeeper with zk-only tests
lhotari Feb 15, 2025
89283e0
Allow filtering providers with TEST_METADATA_PROVIDERS env variable
lhotari Feb 15, 2025
4d833fe
Get rid of reflection in MockZooKeeper
lhotari Feb 15, 2025
cb32580
Handle any exception in multi ops
lhotari Feb 15, 2025
9d1c994
Fix checkstyle
lhotari Feb 15, 2025
85e217c
Log errors in MockZooKeeper multi
lhotari Feb 15, 2025
6e2175b
Fix error handling in MockZooKeeper multi
lhotari Feb 15, 2025
4a5a492
Consistently trigger watches in executor
lhotari Feb 15, 2025
5bd681a
Improve error logging in MockZooKeeper multi
lhotari Feb 15, 2025
4c27282
Always use a single threaded executor
lhotari Feb 15, 2025
be6a424
Replace locks in MockZooKeeper with single threaded execution
lhotari Feb 15, 2025
ccf1371
Improve solution to run pulsar-metadata tests since specific providers
lhotari Feb 15, 2025
3994410
Shutdown outside of executor thread
lhotari Feb 15, 2025
becc221
Fix hasChildren in MockZooKeeper
lhotari Feb 15, 2025
cfc2467
Pass path in NoNodeException
lhotari Feb 15, 2025
18ab5f3
Add logging to see what happens
lhotari Feb 15, 2025
4033fc3
Increase timeout to make the test pass on MockZooKeeper
lhotari Feb 15, 2025
02d01e3
Reduce logging in MockZooKeeper for multiops
lhotari Feb 15, 2025
3852360
Disable invalid test BrokerServiceLookupTest.testLookupConnectionNotC…
lhotari Feb 15, 2025
28bb928
Optimize MockZooKeeper getChildren
lhotari Feb 15, 2025
7d833f8
Fix some NPEs at startup when LeaderElectionService isn't yet available
lhotari Feb 17, 2025
87fad55
Add better exception message
lhotari Feb 17, 2025
6db5a4d
Fix exceptions caused by invalid bundle keys
lhotari Feb 17, 2025
5212fa2
Allow proper shutdown of BrokerService in BrokerServiceLookupTest.tes…
lhotari Feb 17, 2025
f598d95
Refactor restarting broker with different config
lhotari Feb 17, 2025
358d057
Improve MockZooKeeper shutdown
lhotari Feb 17, 2025
c3f24b1
Remove unnecessary parameter
lhotari Feb 17, 2025
9b58676
Reject tasks if stopped
lhotari Feb 17, 2025
70f6321
Improve exception message
lhotari Feb 17, 2025
2c45034
Fix shutdown
lhotari Feb 17, 2025
a631efa
Avoid unnecessary broker restarts in BrokerServiceLookupTest
lhotari Feb 17, 2025
2c7b042
Properly implement reusing TestZKServer in another context
lhotari Feb 17, 2025
e219043
Remove unnecessary synchronization
lhotari Feb 17, 2025
aa6274f
Fix parent handling
lhotari Feb 17, 2025
d0a0505
Refactor
lhotari Feb 17, 2025
4ceecad
Fix pulsar-metadata test resource leaks
lhotari Feb 17, 2025
06e745b
Fix one more leak
lhotari Feb 17, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Avoid unnecessary broker restarts in BrokerServiceLookupTest
  • Loading branch information
lhotari committed Feb 17, 2025
commit a631efaa876cab9b3f6aec6605a64fdcccaf3aca
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ public void applyTestName(Method method) {
testName = method.getName() + " with " + (useTestZookeeper ? "TestZKServer" : "MockZooKeeper");
}

@BeforeMethod
@BeforeMethod(dependsOnMethods = "setTestMethodName")
@Override
protected void setup() throws Exception {
conf.setDefaultNumberOfNamespaceBundles(1);
Expand All @@ -156,6 +156,38 @@ protected void setup() throws Exception {
producerBaseSetup();
}

@Override
protected void doInitConf() throws Exception {
super.doInitConf();
switch (methodName) {
case "testMultipleBrokerDifferentClusterLookup" -> {
conf.setAuthenticationEnabled(true);
}
case "testWebserviceServiceTls" -> {
// broker1 with tls enabled
conf.setBrokerServicePortTls(Optional.of(0));
conf.setWebServicePortTls(Optional.of(0));
conf.setTlsTrustCertsFilePath(CA_CERT_FILE_PATH);
conf.setTlsRequireTrustedClientCertOnConnect(true);
conf.setTlsCertificateFilePath(BROKER_CERT_FILE_PATH);
conf.setTlsKeyFilePath(BROKER_KEY_FILE_PATH);
conf.setNumExecutorThreadPoolSize(5);
// Not in use, and because TLS is not configured, it will fail to start
conf.setSystemTopicEnabled(false);
}
case "testSkipSplitBundleIfOnlyOneBroker" -> {
conf.setDefaultNumberOfNamespaceBundles(1);
conf.setLoadBalancerNamespaceBundleMaxTopics(1);
conf.setLoadManagerClassName(ModularLoadManagerImpl.class.getName());
}
case "testPartitionedMetadataWithDeprecatedVersion" -> {
conf.setBrokerServicePortTls(Optional.empty());
conf.setWebServicePortTls(Optional.empty());
conf.setClientLibraryVersionCheckEnabled(true);
}
}
}

@AfterMethod(alwaysRun = true)
@Override
protected void cleanup() throws Exception {
Expand Down Expand Up @@ -407,12 +439,6 @@ public void testMultipleBrokerDifferentClusterLookup() throws Exception {
@Cleanup
PulsarClient pulsarClient2 = PulsarClient.builder().serviceUrl(brokerServiceUrl.toString()).build();

// restart broker with authorization enabled: it initialize AuthorizationService
restartBroker(conf -> {
// enable authorization: so, broker can validate cluster and redirect if finds different cluster
conf.setAuthorizationEnabled(true);
});

LoadManager loadManager2 = spy(pulsar2.getLoadManager().get());
Field loadManagerField = NamespaceService.class.getDeclaredField("loadManager");
loadManagerField.setAccessible(true);
Expand Down Expand Up @@ -449,10 +475,6 @@ public void testMultipleBrokerDifferentClusterLookup() throws Exception {
consumer.acknowledgeCumulative(msg);
consumer.close();
producer.close();

// disable authorization
pulsar.getConfiguration().setAuthorizationEnabled(false);
loadManager2 = null;
}

/**
Expand Down Expand Up @@ -570,18 +592,6 @@ public void testWebserviceServiceTls() throws Exception {
PulsarTestContext pulsarTestContext2 = createAdditionalPulsarTestContext(conf2);
PulsarService pulsar2 = pulsarTestContext2.getPulsarService();

restartBroker(conf -> {
// restart broker1 with tls enabled
conf.setBrokerServicePortTls(Optional.of(0));
conf.setWebServicePortTls(Optional.of(0));
conf.setTlsTrustCertsFilePath(CA_CERT_FILE_PATH);
conf.setTlsRequireTrustedClientCertOnConnect(true);
conf.setTlsCertificateFilePath(BROKER_CERT_FILE_PATH);
conf.setTlsKeyFilePath(BROKER_KEY_FILE_PATH);
conf.setNumExecutorThreadPoolSize(5);
// Not in use, and because TLS is not configured, it will fail to start
conf.setSystemTopicEnabled(false);
});
pulsar.getLoadManager().get().writeLoadReportOnZookeeper();
pulsar2.getLoadManager().get().writeLoadReportOnZookeeper();

Expand Down Expand Up @@ -785,11 +795,6 @@ public void testModularLoadManagerSplitBundle() throws Exception {
conf2.setLoadBalancerAutoUnloadSplitBundlesEnabled(true);
conf2.setLoadBalancerNamespaceBundleMaxTopics(1);

// configure broker-1 with ModularLoadManager
restartBroker(conf -> {
conf.setLoadManagerClassName(ModularLoadManagerImpl.class.getName());
});

@Cleanup
PulsarTestContext pulsarTestContext2 = createAdditionalPulsarTestContext(conf2);
PulsarService pulsar2 = pulsarTestContext2.getPulsarService();
Expand Down Expand Up @@ -907,12 +912,6 @@ public void testSkipSplitBundleIfOnlyOneBroker() throws Exception {
final String topicName1 = BrokerTestUtil.newUniqueName("persistent://" + namespace + "/tp_");
final String topicName2 = BrokerTestUtil.newUniqueName("persistent://" + namespace + "/tp_");
try {
// configure broker with ModularLoadManager.
restartBroker(conf -> {
conf.setDefaultNumberOfNamespaceBundles(1);
conf.setLoadBalancerNamespaceBundleMaxTopics(1);
conf.setLoadManagerClassName(ModularLoadManagerImpl.class.getName());
});
final ModularLoadManagerWrapper modularLoadManagerWrapper =
(ModularLoadManagerWrapper) pulsar.getLoadManager().get();
final ModularLoadManagerImpl modularLoadManager =
Expand Down Expand Up @@ -1065,12 +1064,6 @@ public void testPartitionedMetadataWithDeprecatedVersion() throws Exception {
admin.namespaces().createNamespace(property + "/" + cluster + "/" + namespace);
admin.topics().createPartitionedTopic(dest.toString(), totalPartitions);

restartBroker(conf -> {
conf.setBrokerServicePortTls(Optional.empty());
conf.setWebServicePortTls(Optional.empty());
conf.setClientLibraryVersionCheckEnabled(true);
});

URI brokerServiceUrl = new URI(pulsar.getSafeWebServiceAddress());

URL url = brokerServiceUrl.toURL();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ public abstract class ProducerConsumerBase extends MockedPulsarServiceBaseTest {
protected String methodName;

@BeforeMethod(alwaysRun = true)
public void beforeMethod(Method m) throws Exception {
public void setTestMethodName(Method m) throws Exception {
methodName = m.getName();
}

Expand Down