The renew grace period is too small to renew locks for a batch of Service Bus Queue messages

[tj---]

• Package Name: azure-servicebus
• Package Version: 7.4.0 / 7.12.2
• Operating System: macOS
• Python Version: 3.8.13

Describe the bug
If we take a batch of messages from the Service Bus Queue, it takes a long time to renew the locks (since there is a sleep of 0.5s between each update). The renewal happens very close to the end of expiry (10s) - so it is a very critical window.
The only solution for a large batch (say of size 400) is to have a ridiculously large number of threads that can renew the locks together.

To Reproduce
Steps to reproduce the behavior:

1. Create a queue (with a lock lifetime of 4m) and push some messages in it.
2. Put the Queue name and Connection string in this sample code and run it. https://gist.github.com/tj---/cf9cec837febea4f9903d987019ba651 [Alternate: with hack]
3. Observe the error logs start showing up after ~4 minutes (only a few locks will be renewed, renewal of the rest will fail)

Expected behavior

1. The renewal of locks should be permitted via a configurable renew_period parameter and soon (eg: I would start renewing the locks 1-2 minutes in advance)
2. It should be possible to increase the lock lifetime at the Queue itself (which is currently a max of 5m)

Screenshots
If applicable, add screenshots to help explain your problem.

1. Logs (with issue):
   redacted_issue_log.txt

2. Logs (With hack in place - increased the _renew_period to 90s and max_workers = 2):
   redacted_ok_log.txt

Additional context
Original reference: #35717 (comment)

[github-actions]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[kristapratico]

@tj--- thanks for the detailed issue, the team will investigate and respond as soon as possible.

[kashifkhan]

Hi @tj--- ,

Apologies on the delay on this one. The reason why this is so small is because the renewal of links happens over the management link versus the regular link. The management link is heavily throttled and deals aggressively with credits conversion and a small renewal grace is meant to keep traffic low, which is why we don't allow modification to that time as well. There is also a risk of keeping that many messages on hand and trying to renew them as you can run out of time, so the user has to keep a balance of messages of hand, size of messages and being able to safely renew them. When testing with messages of varying sizes we got variable results.

So there are a few options that can be tried:

• Renewing fewer messages at a time
• Trying the async client as that will be able to handle more messages.
• Implementing a custom Autolock renew and using that instead, as the autolock renewer code uses this public API on the receiver renew_message_lock.

As for allowing the queue lock time to be greater than 5 minutes, that would be on the service team as they would be able to address that.

[github-actions]

Hi @tj---. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.