Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix][broker][branch-3.1] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (#21213) #21314

Merged
merged 1 commit into from
Oct 18, 2023
Merged

[fix][broker][branch-3.1] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (#21213) #21314

merged 1 commit into from
Oct 18, 2023

Conversation

Demogorgon314
Copy link
Member

(cherry picked from commit f85e0dc)

Motivation

Currently, if the cluster has multiple brokers, and the cluster is doing rolling restart, the heartbeat namespace topic's lookup result might be wrong, because the ExtensibleLoadManagerImpl does not check the heartbeat and SLA namespace bundle lookup candidate broker to let them own by the specified broker.

This ownership selection is wrong:

2023-09-20T08:27:32,188+0000 [ForkJoinPool.commonPool-worker-1] INFO  org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl - Selected new owner broker: broker-2:8080 for bundle: pulsar/broker-0:8080/0x00000000_0xffffffff.

After the ownership assignment, the broker-0 will fail to start.

pulsar-broker 2023-09-20T08:27:54,707+0000 [main] INFO  org.apache.pulsar.broker.loadbalance.extensions.ExtensibleLoadManagerImpl - Try acquiring ownership for bundle: pulsar/broker-0:8080/0x00000000_0xffffffff - broker-0:8080.
pulsar-broker 2023-09-20T08:27:54,707+0000 [main] ERROR org.apache.pulsar.broker.namespace.NamespaceService - namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:400) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker 2023-09-20T08:27:54,710+0000 [main] ERROR org.apache.pulsar.broker.PulsarService - Failed to start Pulsar service: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker org.apache.pulsar.broker.PulsarServerException: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:403) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker Caused by: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:400) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     ... 4 more
pulsar-broker 2023-09-20T08:27:54,710+0000 [main] ERROR org.apache.pulsar.PulsarBrokerStarter - Failed to start pulsar service.
pulsar-broker org.apache.pulsar.broker.PulsarServerException: org.apache.pulsar.broker.PulsarServerException: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker     at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:938) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.PulsarBrokerStarter$BrokerStarter.start(PulsarBrokerStarter.java:276) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.PulsarBrokerStarter.main(PulsarBrokerStarter.java:356) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker Caused by: org.apache.pulsar.broker.PulsarServerException: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:403) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     ... 2 more
pulsar-broker Caused by: java.lang.IllegalStateException: namespace already owned by other broker : ns=pulsar/broker-0:8080 expected=pulsar://broker-0:6650 actual=pulsar://broker-2:6650
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerNamespace(NamespaceService.java:400) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.namespace.NamespaceService.registerBootstrapNamespaces(NamespaceService.java:343) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     at org.apache.pulsar.broker.PulsarService.start(PulsarService.java:863) ~[io.streamnative-pulsar-broker-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
pulsar-broker     ... 2 more
pulsar-broker 2023-09-20T08:27:54,711+0000 [main] WARN  org.apache.pulsar.common.util.ShutdownUtil - Triggering immediate shutdown of current process with status 1

Modifications

  • Fix lookup heartbeat and sla namespace bundle when using extensible load manager
  • When updating the topK bundles, filter out the heartbeat and SLA namespace bundles.
  • Skip override orphan heartbeat namespace bundle

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

@Demogorgon314
[fix][broker] Fix lookup heartbeat and sla namespace bundle when usin…
f242022 
…g extensible load manager (apache#21213)

(cherry picked from commit f85e0dc)
@Demogorgon314 Demogorgon314 self-assigned this Oct 8, 2023
@Demogorgon314 Demogorgon314 changed the title [fix][broker] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (#21213) [branch-3.1][fix][broker] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (#21213) Oct 8, 2023
@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Oct 8, 2023
@Demogorgon314 Demogorgon314 changed the title [branch-3.1][fix][broker] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (#21213) [fix][broker][branch-3.1] Fix lookup heartbeat and sla namespace bundle when using extensible load manager (#21213) Oct 11, 2023
@Demogorgon314 Demogorgon314 reopened this Oct 16, 2023
@Demogorgon314 Demogorgon314 merged commit 0454410 into apache:branch-3.1 Oct 18, 2023
@Demogorgon314 Demogorgon314 deleted the Demogorgon314/cherry-pick-21213-to-branch-3.1 branch October 18, 2023 01:04
@Technoboy- Technoboy- mentioned this pull request Oct 19, 2023
Merged
4 tasks
nikhil-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
@Demogorgon314 @nikhil-ctds
[fix][broker][branch-3.1] Fix lookup heartbeat and sla namespace bund…
5f358ec 
…le when using extensible load manager (apache#21213) (apache#21314)
srinath-ctds pushed a commit to datastax/pulsar that referenced this pull request Dec 20, 2023
@Demogorgon314 @srinath-ctds
[fix][broker][branch-3.1] Fix lookup heartbeat and sla namespace bund…
f027857 
…le when using extensible load manager (apache#21213) (apache#21314)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Feb 29, 2024
@Demogorgon314 @mukesh-ctds
[fix][broker][branch-3.1] Fix lookup heartbeat and sla namespace bund…
0e6c049 
…le when using extensible load manager (apache#21213) (apache#21314)

(cherry picked from commit 0454410)
mukesh-ctds pushed a commit to datastax/pulsar that referenced this pull request Mar 6, 2024
@Demogorgon314 @mukesh-ctds
[fix][broker][branch-3.1] Fix lookup heartbeat and sla namespace bund…
3e1825d 
…le when using extensible load manager (apache#21213) (apache#21314)

(cherry picked from commit 0454410)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gaoran10 gaoran10 gaoran10 approved these changes

@Technoboy- Technoboy- Awaiting requested review from Technoboy-

@heesung-sn heesung-sn Awaiting requested review from heesung-sn

@mattisonchao mattisonchao Awaiting requested review from mattisonchao

Assignees

@Demogorgon314 Demogorgon314

Labels
cherry-picked/branch-3.1 doc-not-needed Your PR changes do not impact docs ready-to-test release/3.1.2
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@Demogorgon314 @gaoran10 @AnonHxy