[Doc] Improve Pulsar Pub/Sub Throttling Doc

[heesung-sn]

Search before asking

• I searched in the issues and found nothing similar.

What issue do you find in Pulsar docs?

The current documentation about Pulsar message pub/sub throttling is insufficient.

What is your suggestion?

• Have a central page about Pulsar message throttling.
• Explain the goal of Pulsar message throttling.
• Explain how and where Pulsar throttling works.
• Explain what configs and admin APIs are available to set throttling.
• Explain how throttling broker configs and admin APIs are working together if they are configured at the same time. (explain config-adminAPI compatibility)
• Explain how throttling works with different features: message batching, compression, chunking, backlog, non-durable topics, partitions, replicas, entry filter, flink, sink/source, functions, offload and etc. (explain feature compatibility)
• Explain how throttling can be monitored
• Give example setups for best throttling practices

Any reference?

https://pulsar.apache.org/docs/2.10.x/admin-api-namespaces/#configure-dispatch-throttling-for-topics

pulsar % grep Throttling conf/broker.conf
topicPublisherThrottlingTickTimeMillis=10
brokerPublisherThrottlingTickTimeMillis=50
brokerPublisherThrottlingMaxMessageRate=0
brokerPublisherThrottlingMaxByteRate=0
subscribeThrottlingRatePerConsumer=0
dispatchThrottlingRateInMsg=0
dispatchThrottlingRateInByte=0
dispatchThrottlingRatePerTopicInMsg=0
dispatchThrottlingRatePerTopicInByte=0
dispatchThrottlingOnBatchMessageEnabled=false
dispatchThrottlingRatePerSubscriptionInMsg=0
dispatchThrottlingRatePerSubscriptionInByte=0
dispatchThrottlingRatePerReplicatorInMsg=0
dispatchThrottlingRatePerReplicatorInByte=0
dispatchThrottlingRateRelativeToPublishRate=false
dispatchThrottlingOnNonBacklogConsumerEnabled=true
dispatchThrottlingForFilteredEntriesEnabled=false

Secondary configs that could be related to throttling behavior:
dispatcherMaxReadBatchSize
avgMessagesPerEntry

Pulsar-admin brokers update-dynamic-config --config dispatcherMaxReadBatchSize —value 100

Some references regarding the above configs:
#5710
#11325
#2977
#754
#4273
#5804
#12294
#17686

Are you willing to submit a PR?

• I'm willing to submit a PR!

[Hongten]

I think this document is very SIMPLE. The document doesn't show us how to verify the throttle, just call the get-... API to show the values which come from the ZK(We can tell users we save these throttling values with the exact path in ZK, right?).

So there are some suggestions you can refer to

• How to set the publish rate? How to verify it(e.g. we can refer to some metrics pulsar_throughput_in)? Based on my test(pulsar-2.10.1), it seems not to work when I set the publish rate(both bytes and message).
• How to set the dispatch rate? How to verify it(e.g. we can refer to some metrics pulsar_throughput_out)?
• What's the difference when we set broker, topic publish/dispatch rate?
• What's the difference between setting publish/dispatch rate and setting a sourcegroup? Please refer to PIP-82

Please refer to the test results - #18699

Thanks
Hongten

[github-actions]

The issue had no activity for 30 days, mark with Stale label.