Closed
Description
- Package Name: azure-servicebus
- Package Version: 7.0.0
- Operating System: Linux Debian/Ubuntu
- Python Version: 3.7.9
- uamqp Version: 1.2.12
Describe the bug
An async permanent service bus receiver is failing due to authorization timeout after some hours of operation.
The below debug logs should indicate that the token refreshing is working:
2020-12-24 06:35:36,199 uamqp.authentication.cbs_auth_async [INFO] Token on connection b'SBReceiver-REDACTED' will expire soon - attempting to refresh.
2020-12-24 06:35:36,199 azure.identity.aio._internal.decorators [INFO] ManagedIdentityCredential.get_token succeeded
But it is not. After a large amount of token refreshing attempts (at ~50ms intervals), the following exception is thrown, and the connection is closed:
2020-12-24 06:45:19,000 azure.servicebus.aio._base_handler_async [ERROR] Unexpected error occurred (TimeoutError('Authorization timeout.')). Handler shutting down.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/azure/servicebus/aio/_base_handler_async.py", line 181, in _do_retryable_operation
return await operation(**kwargs)
File "/usr/local/lib/python3.7/site-packages/azure/servicebus/aio/_servicebus_receiver_async.py", line 343, in _receive
receiving = await amqp_receive_client.do_work_async()
File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 378, in do_work_async
if not await self.client_ready_async():
File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 360, in client_ready_async
if not await self.auth_complete_async():
File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 346, in auth_complete_async
raise TimeoutException("Authorization timeout.")
TimeoutError: Authorization timeout.
To Reproduce
Steps to reproduce the behavior:
- Create an async receiver using DefaultAzureCredential
- Start an indefinite iteration to create a permanent receiver waiting for messages
- After some hours of operation, the program will fail due to authorization timeout
Expected behavior
Authorization refreshing is managed successfully by the library.
Code
Basic code to reproduce the issue:
from datetime import datetime
from asyncio import get_event_loop, sleep
from azure.identity.aio import DefaultAzureCredential
from azure.servicebus.aio import ServiceBusClient
from azure.servicebus import ServiceBusReceiveMode
import logging
logging.basicConfig(level=logging.DEBUG, format="%(asctime)-15s %(name)s [%(levelname)s] %(message)s")
SB_NAMESPACE_FQDN = "..."
QUEUE_NAME = "..."
async def receive_forever():
attempt = 0
while True:
start = datetime.now()
try:
attempt += 1
logging.info(f"== Starting attempt={attempt}")
async with DefaultAzureCredential(exclude_environment_credential=True) as credential:
service_bus_client = ServiceBusClient(SB_NAMESPACE_FQDN, credential, logging_enable=True)
async with service_bus_client:
queue_receiver = service_bus_client.get_queue_receiver(
QUEUE_NAME,
receive_mode=ServiceBusReceiveMode.RECEIVE_AND_DELETE,
)
async with queue_receiver:
while True:
messages = await queue_receiver.receive_messages(max_message_count=1, max_wait_time=30)
if messages:
logging.debug(f"Received: {messages[0]}")
except Exception:
logging.exception(f"== Running for {datetime.now()-start}")
await sleep(10)
if __name__ == "__main__":
loop = get_event_loop()
loop.run_until_complete(receive_forever())
Please find attached an reduced output log.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Metadata
Assignees
Labels
This issue points to a problem in the data-plane of the library.This issue requires a change to an existing behavior in the product in order to be resolved.Issues that are reported by GitHub users external to the Azure organization.Workflow: More information is needed from author to address the issue.There has been no recent activity on this issue.The issue doesn't require a change to the product in order to be resolved. Most issues start as that