[improve][pip] PIP-434: Expose Netty channel configuration WRITE_BUFFER_WATER_MARK to pulsar conf and pause receive requests when channel is unwritable

[poorbarcode]

Motivation & Modifications

• Discussion
• Vote

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: x

[lhotari]

besides configuring the high water mark and the low water mark, there should be a way to disable the action (ServerCnxThrottleTracker's
incrementThrottleCount/decrementThrottleCount) that is executed in the channelWritabilityChanged callback method.

[poorbarcode]

Added a new param pulsarChannelPauseReceivingRequestsIfUnwritable=false

[lhotari]

The new parameter resolves this. It would be also be useful to describe the implementation detail of handling the action in channelWritabilityChanged and the usage of ServerCnxThrottleTracker in this action (somewhere in the PIP document).

[lhotari]

Throttling incoming requests for individual channels (connections) isn't a very effective way to solve this problem. However, I do support adding a way to configure WRITE_BUFFER_WATER_MARK parameters for the Netty channel.

There will be a negative impact to performance in cases where the write buffer size exceeds due to other reasons. This is due to the fact that there will be less pipelining (parallel processing) of incoming requests when incoming requests are throttled whenever the channel isn't writable. For example, this will negatively impact the latency of publish (send) requests while consumers are dispatching messages (and the messages in the write buffer cause the size of the write buffer to exceed the high water mark).

I have explained this in the mailing list thread, but the arguments seem to have been ignored there, so let's handle the argument here.

[poorbarcode]

I have explained this in the mailing list thread, but the arguments seem to have been ignored there, so let's handle the argument here.

It was not ignored. I have answered it in the mailing list thread

Once the channel is not writable, all requests that are sent to this
channel can not receive a reply anymore because the response can not be
written out. The results are the same; the clients receive replies delayed.
To improve performance, users should consider using more channels; in other
words, they can set a bigger connectionsPerBroker or separate clients.

---

For example, this will negatively impact the latency of publish (send) requests while consumers are dispatching messages (and the messages in the write buffer cause the size of the write buffer to exceed the high water mark).

Yes, it will. However, once the channel is not writable, the producer cannot receive a send response, which also delays message publishing. And we have exposed the 3 configurations to users, they can adjust them if needed

[lhotari]

Yes, it will. However, once the channel is not writable, the producer cannot receive a send response, which also delays message publishing.

Yes, I agree that from the client's perspective, it would be delayed since it wouldn't receive a response immediately. However, there's an important difference here. When incoming send requests are continued to be processed, but the messages will get persisted and processed by the broker and therefore pipelining is happening. Allowing pipelining is important for Pulsar's performance, where it is common that the same connection is shared for multiple purposes (consumers, producers, etc.).

And we have exposed the 3 configurations to users, they can adjust them if needed

Sure, users can tune the parameters and that makes sense. However, the default values and configuration examples should contain useful examples. The default high watermark value should be relatively high so that it wouldn't impact use cases where there's consumers and producers sharing the same connection.