Flaky-test: AdminApi2Test.resetClusters

[lhotari]

Search before asking

• I searched in the issues and found nothing similar.

Example failure

https://github.com/apache/pulsar/actions/runs/5056774399/jobs/9075132818?pr=20372#step:10:1446

Exception stacktrace

  Error:  Tests run: 82, Failures: 1, Errors: 0, Skipped: 55, Time elapsed: 660.065 s <<< FAILURE! - in org.apache.pulsar.broker.admin.AdminApi2Test
  Error:  org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters  Time elapsed: 2.209 s  <<< FAILURE!
  org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: 
  
   --- An unexpected error occurred in the server ---
  
  Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
  
  Stacktrace:
  
  org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
  	at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
  	at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
  	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
  	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
  	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  	at java.base/java.lang.Thread.run(Thread.java:833)
  Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
  	... 14 more
  
  	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
  	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
  	at org.apache.pulsar.client.admin.internal.TenantsImpl.deleteTenant(TenantsImpl.java:93)
  	at org.apache.pulsar.broker.admin.AdminApi2Test.resetClusters(AdminApi2Test.java:181)
  	at jdk.internal.reflect.GeneratedMethodAccessor548.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
  	at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
  	at org.testng.TestRunner.run(TestRunner.java:602)
  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:431)
  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:391)
  	at org.testng.SuiteRunner.run(SuiteRunner.java:330)
  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:95)
  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1256)
  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1176)
  	at org.testng.TestNG.runSuites(TestNG.java:1099)
  	at org.testng.TestNG.run(TestNG.java:1067)
  	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:155)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:102)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeLazy(TestNGDirectoryTestSuite.java:117)
  	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:86)
  	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:137)
  	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
  	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
  	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
  	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
  	Suppressed: org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: 
   --- An unexpected error occurred in the server ---
  
  Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
  
  Stacktrace:
  
  org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
  	at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
  	at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
  	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
  	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
  	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  	at java.base/java.lang.Thread.run(Thread.java:833)
  Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
  	... 14 more
  
  		at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:272)
  		at org.apache.pulsar.client.admin.internal.BaseResource$4.failed(BaseResource.java:237)
  		at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
  		at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
  		at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
  		at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
  		at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
  		at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
  		at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
  		at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
  		at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
  		at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
  		at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
  		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
  		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
  		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
  		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
  		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
  		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
  		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  		at java.base/java.lang.Thread.run(Thread.java:833)
  	Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}
  		at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
  		at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
  		... 54 more
  Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}]

Are you willing to submit a PR?

• I'm willing to submit a PR!

[michaeljmarshall]

Another: https://github.com/apache/pulsar/actions/runs/5059985368/jobs/9085290991?pr=20377#step:10:1656

[lhotari]

Error:  Tests run: 104, Failures: 1, Errors: 0, Skipped: 61, Time elapsed: 465.204 s <<< FAILURE! - in org.apache.pulsar.client.api.SimpleProducerConsumerTest
  Error:  org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest  Time elapsed: 5.1 s  <<< FAILURE!
  org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException: 
  
   --- An unexpected error occurred in the server ---
  
  Message: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
  
  Stacktrace:
  
  org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)
  	at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)
  	at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)
  	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)
  	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)
  	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
  	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
  	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
  	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  	at java.base/java.lang.Thread.run(Thread.java:833)
  Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property
  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:132)
  	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
  	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)
  	... 14 more
  
  	at org.apache.pulsar.client.admin.PulsarAdminException.wrap(PulsarAdminException.java:252)
  	at org.apache.pulsar.client.admin.internal.BaseResource.sync(BaseResource.java:352)
  	at org.apache.pulsar.client.admin.internal.TenantsImpl.deleteTenant(TenantsImpl.java:93)
  	at org.apache.pulsar.client.api.SimpleProducerConsumerTest.rest(SimpleProducerConsumerTest.java:149)
  	at jdk.internal.reflect.GeneratedMethodAccessor338.invoke(Unknown Source)
  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:139)
  	at org.testng.internal.invokers.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:69)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurationMethod(ConfigInvoker.java:361)
  	at org.testng.internal.invokers.ConfigInvoker.invokeConfigurations(ConfigInvoker.java:296)
  	at org.testng.internal.invokers.TestInvoker.runConfigMethods(TestInvoker.java:823)
  	at org.testng.internal.invokers.TestInvoker.runAfterConfigurations(TestInvoker.java:792)
  	at org.testng.internal.invokers.TestInvoker.invokeMethod(TestInvoker.java:768)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethod(TestInvoker.java:221)
  	at org.testng.internal.invokers.MethodRunner.runInSequence(MethodRunner.java:50)
  	at org.testng.internal.invokers.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:969)
  	at org.testng.internal.invokers.TestInvoker.invokeTestMethods(TestInvoker.java:194)
  	at org.testng.internal.invokers.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:148)
  	at org.testng.internal.invokers.TestMethodWorker.run(TestMethodWorker.java:128)
  	at java.base/java.util.ArrayList.forEach(ArrayList.java:[1511](https://github.com/apache/pulsar/actions/runs/5068165357/jobs/9100493728#step:10:1512))
  	at org.testng.TestRunner.privateRun(TestRunner.java:829)
  	at org.testng.TestRunner.run(TestRunner.java:602)
  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:437)
  		at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
  		at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
  		at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
  		at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
  		at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
  		at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
  		at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
  		at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:254)
  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
  		at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:296)
  		at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)
  		at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)
  		at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
  		at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147)
  		at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
  		at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
  		at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
  		at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
  		at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
  		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
  		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
  		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
  		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
  		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
  		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
  		at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
  		at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
  		at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
  		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
  		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  		at java.base/java.lang.Thread.run(Thread.java:833)
  	Caused by: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}
  		at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:960)
  		at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
  		... 54 more
  Caused by: [CIRCULAR REFERENCE: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/my-property\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}]

https://github.com/apache/pulsar/actions/runs/5068165357/jobs/9100493728#step:10:1495

[michaeljmarshall]

I downloaded the logs for this test, and here is some additional context for the error:

2023-05-24T15:30:44,959 - INFO  - [configuration-metadata-store-2710-1:ResourceGroupNamespaceConfigListener@147] - Metadata store notification: Path /admin/policies/prop-xyz/ns1, Type Deleted
2023-05-24T15:30:44,960 - INFO  - [metadata-store-2708-1:Namespaces@195] - [null] Successful delete namespace ns1
2023-05-24T15:30:44,960 - INFO  - [metadata-store-2708-1:Slf4jRequestLogWriter@62] - 127.0.0.1 - - [24/May/2023:15:30:44 +0000] "DELETE /admin/v2/namespaces/prop-xyz/ns1?force=true HTTP/1.1" 204 0 "-" "Pulsar-Java-v3.1.0-SNAPSHOT" 30
2023-05-24T15:30:44,961 - INFO  - [metadata-store-2708-1:NamespaceBundleFactory@183] - Policy updated for namespace Optional[prop-xyz/ns1], refreshing the bundle cache.
2023-05-24T15:30:44,970 - INFO  - [metadata-store-2708-1:TenantsBase@205] - [null] Deleted tenant prop-xyz
2023-05-24T15:30:44,970 - INFO  - [metadata-store-2708-1:Slf4jRequestLogWriter@62] - 127.0.0.1 - - [24/May/2023:15:30:44 +0000] "DELETE /admin/v2/tenants/prop-xyz?force=true HTTP/1.1" 204 0 "-" "Pulsar-Java-v3.1.0-SNAPSHOT" 5
2023-05-24T15:30:44,973 - INFO  - [pulsar-web-2736-1:Slf4jRequestLogWriter@62] - 127.0.0.1 - - [24/May/2023:15:30:44 +0000] "GET /admin/v2/namespaces/prop-xyz2 HTTP/1.1" 200 2 "-" "Pulsar-Java-v3.1.0-SNAPSHOT" 1
2023-05-24T15:30:44,982 - ERROR - [metadata-store-2708-1:TenantsBase@210] - [null] Failed to delete tenant prop-xyz2
org.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006) ~[testmocks-3.1.0-SNAPSHOT.jar:3.8.1]
	at org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191) ~[testmocks-3.1.0-SNAPSHOT.jar:3.8.1]
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:132) ~[zookeeper-3.8.1.jar:3.8.1]
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.8.1.jar:3.8.1]
	at org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466) ~[pulsar-metadata-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
	... 14 more
2023-05-24T15:30:44,985 - WARN  - [AsyncHttpClient-2797-1:BaseResource$4@236] - [http://localhost:42995/admin/v2/tenants/prop-xyz2?force=true] Failed to perform http delete request: javax.ws.rs.InternalServerErrorException: HTTP 500 {"reason":"\n --- An unexpected error occurred in the server ---\n\nMessage: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\nStacktrace:\n\norg.apache.pulsar.metadata.api.MetadataStoreException: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:476)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.handleDeleteResult(ZKMetadataStore.java:304)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.lambda$batchOperation$5(ZKMetadataStore.java:216)\n\tat org.apache.zookeeper.MockZooKeeper.multi(MockZooKeeper.java:1006)\n\tat org.apache.zookeeper.MockZooKeeperSession.multi(MockZooKeeperSession.java:191)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.batchOperation(ZKMetadataStore.java:190)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.internalBatchOperation(AbstractBatchedMetadataStore.java:184)\n\tat org.apache.pulsar.metadata.impl.batching.AbstractBatchedMetadataStore.flush(AbstractBatchedMetadataStore.java:124)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.apache.zookeeper.KeeperException$NotEmptyException: KeeperErrorCode = Directory not empty for /managed-ledgers/prop-xyz2\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:132)\n\tat org.apache.zookeeper.KeeperException.create(KeeperException.java:54)\n\tat org.apache.pulsar.metadata.impl.ZKMetadataStore.getException(ZKMetadataStore.java:466)\n\t... 14 more\n"}
2023-05-24T15:30:44,987 - INFO  - [metadata-store-2708-1:Slf4jRequestLogWriter@62] - 127.0.0.1 - - [24/May/2023:15:30:44 +0000] "DELETE /admin/v2/tenants/prop-xyz2?force=true HTTP/1.1" 500 2240 "-" "Pulsar-Java-v3.1.0-SNAPSHOT" 12

Note that resetClusters is not a test, but is a lifecycle method that is called after every test run:

https://github.com/apache/pulsar/blob/e956db729f5098ff319237fd4220dbfb234c1b18/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApi2Test.java#L172C1-L190

    @AfterMethod(alwaysRun = true)
    public void resetClusters() throws Exception {
        pulsar.getConfiguration().setForceDeleteTenantAllowed(true);
        pulsar.getConfiguration().setForceDeleteNamespaceAllowed(true);
        for (String tenant : admin.tenants().getTenants()) {
            for (String namespace : admin.namespaces().getNamespaces(tenant)) {
                deleteNamespaceWithRetry(namespace, true, admin, pulsar,
                        mockPulsarSetup.getPulsar());
            }
            admin.tenants().deleteTenant(tenant, true);
        }

        for (String cluster : admin.clusters().getClusters()) {
            admin.clusters().deleteCluster(cluster);
        }

        resetConfig();
        setupClusters();
    }

[michaeljmarshall]

Looks like the namespace deletion was also an issue, and that was "solved" by adding retry logic in the aforementioned code. See #18220.

[michaeljmarshall]

I wonder if there is a race condition for enabling force deletion of tenants?

Edit: the answer is no. We'd see a different error.

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TenantsBase.java

Lines 244 to 247 in dc29f3a

	if (!pulsar().getConfiguration().isForceDeleteTenantAllowed()) {
	return FutureUtil.failedFuture(
	new RestException(Status.METHOD_NOT_ALLOWED, "Broker doesn't allow forced deletion of tenants"));
	}

[michaeljmarshall]

The stack trace is similar to #12553, which hints at issues with the metadata model being eventually consistent (which would explain why retries on the namespace deletion resolved the flakiness in the test).

[michaeljmarshall]

Interestingly, there are mailing list threads about this subject from a while ago: https://lists.apache.org/thread/glc0c4dbcpv2sfccmdz5hm5tdqk4ms75

That thread makes me wonder if the tenant deletion is issuing multiple requests to delete child nodes and then issuing requests to delete parent nodes without waiting.

[michaeljmarshall]

It seems likely that something is incorrectly coordinated. Further, it looks like tenant deletion can easily lead to corrupted metadata. The logic is just chained futures without error handling:

pulsar/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/TenantsBase.java

Lines 224 to 241 in dc29f3a

	protected CompletableFuture<Void> internalDeleteTenantAsync(String tenant) {
	return tenantResources().tenantExistsAsync(tenant)
	.thenAccept(exists -> {
	if (!exists) {
	throw new RestException(Status.NOT_FOUND, "Tenant doesn't exist");
	}
	})
	.thenCompose(__ -> hasActiveNamespace(tenant))
	.thenCompose(__ -> tenantResources().deleteTenantAsync(tenant))
	.thenCompose(__ -> pulsar().getPulsarResources().getTopicResources().clearTenantPersistence(tenant))
	.thenCompose(__ -> pulsar().getPulsarResources().getNamespaceResources().deleteTenantAsync(tenant))
	.thenCompose(__ -> pulsar().getPulsarResources().getNamespaceResources()
	.getPartitionedTopicResources().clearPartitionedTopicTenantAsync(tenant))
	.thenCompose(__ -> pulsar().getPulsarResources().getLocalPolicies()
	.deleteLocalPoliciesTenantAsync(tenant))
	.thenCompose(__ -> pulsar().getPulsarResources().getNamespaceResources()
	.deleteBundleDataTenantAsync(tenant));
	}

[michaeljmarshall]

Here is the rest of my error log file:
org.apache.pulsar.broker.admin.AdminApi2Test-output.txt

With additional inspection, it seems like part of the problem could easily be system topics. I see issues related to __change-events topic:

2023-05-24T15:30:44,862 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentSubscription@880] - [persistent://prop-xyz2/ns1/__change_events-partition-11][multiTopicsReader-1b06c712cc] Successfully closed subscription [NonDurableCursorImpl{ledger=prop-xyz2/ns1/persistent/__change_events-partition-11, ackPos=66:-1, readPos=66:0}]
2023-05-24T15:30:44,862 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentSubscription@312] - [persistent://prop-xyz2/ns1/__change_events-partition-11][multiTopicsReader-1b06c712cc] Successfully closed dispatcher for reader
2023-05-24T15:30:44,862 - ERROR - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentTopic@948] - [persistent://prop-xyz2/ns1/__change_events-partition-11] Failed to create subscription: multiTopicsReader-1b06c712cc
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor 
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1189) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$20(PersistentTopic.java:890) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$22(PersistentTopic.java:885) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.internalSubscribe(PersistentTopic.java:806) ~[classes/:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:780) ~[classes/:?]
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$14(ServerCnx.java:1239) ~[classes/:?]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$closeLedger$42(BookkeeperSchemaStorage.java:619) ~[classes/:?]
	at org.apache.bookkeeper.client.PulsarMockLedgerHandle.lambda$asyncClose$1(PulsarMockLedgerHandle.java:96) ~[testmocks-3.1.0-SNAPSHOT.jar:4.16.1]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[bookkeeper-common-4.16.1.jar:4.16.1]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[bookkeeper-common-4.16.1.jar:4.16.1]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor 
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$19(PersistentTopic.java:914) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
	... 22 more
2023-05-24T15:30:44,864 - WARN  - [PulsarTestContext-executor-OrderedExecutor-0-0:ServerCnx@1282] - [/127.0.0.1:57364][persistent://prop-xyz2/ns1/__change_events-partition-11][multiTopicsReader-1b06c712cc] Failed to create consumer: consumerId=11, Connection was closed while the opening the cursor 
2023-05-24T15:30:44,864 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentTopic@1033] - [persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Creating non-durable subscription at msg id -1:-1:-1:-1 - {}
2023-05-24T15:30:44,864 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:NonDurableCursorImpl@66] - [prop-xyz2/ns1/persistent/__change_events-partition-4] Created non-durable cursor read-position=67:0 mark-delete-position=67:-1
2023-05-24T15:30:44,864 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:ManagedLedgerImpl@1130] - [prop-xyz2/ns1/persistent/__change_events-partition-4] Opened new cursor: NonDurableCursorImpl{ledger=prop-xyz2/ns1/persistent/__change_events-partition-4, ackPos=67:-1, readPos=67:0}
2023-05-24T15:30:44,865 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:AbstractDispatcherSingleActiveConsumer@211] - Removing consumer Consumer{subscription=PersistentSubscription{topic=persistent://prop-xyz2/ns1/__change_events-partition-4, name=multiTopicsReader-1b06c712cc}, consumerId=4, consumerName=ae38e, address=/127.0.0.1:57364}
2023-05-24T15:30:44,865 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentSubscription@880] - [persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Successfully closed subscription [NonDurableCursorImpl{ledger=prop-xyz2/ns1/persistent/__change_events-partition-4, ackPos=67:-1, readPos=67:0}]
2023-05-24T15:30:44,865 - INFO  - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentSubscription@312] - [persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Successfully closed dispatcher for reader
2023-05-24T15:30:44,865 - ERROR - [PulsarTestContext-executor-OrderedExecutor-0-0:PersistentTopic@948] - [persistent://prop-xyz2/ns1/__change_events-partition-4] Failed to create subscription: multiTopicsReader-1b06c712cc
java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor 
	at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:368) ~[?:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1189) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$20(PersistentTopic.java:890) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$22(PersistentTopic.java:885) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
	at java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2309) ~[?:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.internalSubscribe(PersistentTopic.java:806) ~[classes/:?]
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.subscribe(PersistentTopic.java:780) ~[classes/:?]
	at org.apache.pulsar.broker.service.ServerCnx.lambda$handleSubscribe$14(ServerCnx.java:1239) ~[classes/:?]
	at java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
	at org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$closeLedger$42(BookkeeperSchemaStorage.java:619) ~[classes/:?]
	at org.apache.bookkeeper.client.PulsarMockLedgerHandle.lambda$asyncClose$1(PulsarMockLedgerHandle.java:96) ~[testmocks-3.1.0-SNAPSHOT.jar:4.16.1]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.safeRunTask(SingleThreadExecutor.java:137) ~[bookkeeper-common-4.16.1.jar:4.16.1]
	at org.apache.bookkeeper.common.util.SingleThreadExecutor.run(SingleThreadExecutor.java:107) ~[bookkeeper-common-4.16.1.jar:4.16.1]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.89.Final.jar:4.1.89.Final]
	at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.pulsar.broker.service.BrokerServiceException: Connection was closed while the opening the cursor 
	at org.apache.pulsar.broker.service.persistent.PersistentTopic.lambda$internalSubscribe$19(PersistentTopic.java:914) ~[classes/:?]
	at java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1187) ~[?:?]
	... 22 more
2023-05-24T15:30:44,866 - WARN  - [PulsarTestContext-executor-OrderedExecutor-0-0:ServerCnx@1282] - [/127.0.0.1:57364][persistent://prop-xyz2/ns1/__change_events-partition-4][multiTopicsReader-1b06c712cc] Failed to create consumer: consumerId=4, Connection was closed while the opening the cursor 
2023-05-24T15:30:44,922 - INFO  - [broker-client-shared-timer-2719-1:ConnectionHandler@124] - [persistent://prop-xyz2/ns1/__change_events-partition-19] [multiTopicsReader-1b06c712cc] Reconnecting after connection was closed
2023-05-24T15:30:44,922 - INFO  - [broker-client-shared-timer-2719-1:ConnectionHandler@68] - [persistent://prop-xyz2/ns1/__change_events-partition-19] [multiTopicsReader-1b06c712cc] Ignoring reconnection request (state: Closed)
2023-05-24T15:30:44,923 - INFO  - [broker-client-shared-timer-2719-1:ConnectionHandler@124] - [persistent://prop-xyz2/ns1/__change_events-partition-17] [multiTopicsReader-1b06c712cc] Reconnecting after connection was closed
2023-05-24T15:30:44,923 - INFO  - [broker-client-shared-timer-2719-1:ConnectionHandler@68] - [persistent://prop-xyz2/ns1/__change_events-partition-17] [multiTopicsReader-1b06c712cc] Ignoring reconnection request (state: Closed)

I wonder if the issue is related to how system topics are deleted?

[michaeljmarshall]

Is it valid to have persistent://prop-xyz2/ns1/__change_events-partition-11 as a persistent partitioned system topic? This was recently fixed for another kind of system topic here #20370 by @heesung-sn.