Skip to content

ServiceBus: Async permanent receiver failing due to authorization timeout #16037

Closed

Description

  • Package Name: azure-servicebus
  • Package Version: 7.0.0
  • Operating System: Linux Debian/Ubuntu
  • Python Version: 3.7.9
  • uamqp Version: 1.2.12

Describe the bug
An async permanent service bus receiver is failing due to authorization timeout after some hours of operation.

The below debug logs should indicate that the token refreshing is working:

2020-12-24 06:35:36,199 uamqp.authentication.cbs_auth_async [INFO] Token on connection b'SBReceiver-REDACTED' will expire soon - attempting to refresh.
2020-12-24 06:35:36,199 azure.identity.aio._internal.decorators [INFO] ManagedIdentityCredential.get_token succeeded

But it is not. After a large amount of token refreshing attempts (at ~50ms intervals), the following exception is thrown, and the connection is closed:

2020-12-24 06:45:19,000 azure.servicebus.aio._base_handler_async [ERROR] Unexpected error occurred (TimeoutError('Authorization timeout.')). Handler shutting down.
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/azure/servicebus/aio/_base_handler_async.py", line 181, in _do_retryable_operation
    return await operation(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/azure/servicebus/aio/_servicebus_receiver_async.py", line 343, in _receive
    receiving = await amqp_receive_client.do_work_async()
  File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 378, in do_work_async
    if not await self.client_ready_async():
  File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 360, in client_ready_async
    if not await self.auth_complete_async():
  File "/usr/local/lib/python3.7/site-packages/uamqp/async_ops/client_async.py", line 346, in auth_complete_async
    raise TimeoutException("Authorization timeout.")
TimeoutError: Authorization timeout.

To Reproduce
Steps to reproduce the behavior:

  1. Create an async receiver using DefaultAzureCredential
  2. Start an indefinite iteration to create a permanent receiver waiting for messages
  3. After some hours of operation, the program will fail due to authorization timeout

Expected behavior
Authorization refreshing is managed successfully by the library.

Code
Basic code to reproduce the issue:

from datetime import datetime
from asyncio import get_event_loop, sleep
from azure.identity.aio import DefaultAzureCredential
from azure.servicebus.aio import ServiceBusClient
from azure.servicebus import ServiceBusReceiveMode

import logging

logging.basicConfig(level=logging.DEBUG, format="%(asctime)-15s %(name)s [%(levelname)s] %(message)s")

SB_NAMESPACE_FQDN = "..."
QUEUE_NAME = "..."


async def receive_forever():
    attempt = 0
    while True:
        start = datetime.now()
        try:
            attempt += 1
            logging.info(f"== Starting attempt={attempt}")

            async with DefaultAzureCredential(exclude_environment_credential=True) as credential:
                service_bus_client = ServiceBusClient(SB_NAMESPACE_FQDN, credential, logging_enable=True)
                async with service_bus_client:
                    queue_receiver = service_bus_client.get_queue_receiver(
                        QUEUE_NAME,
                        receive_mode=ServiceBusReceiveMode.RECEIVE_AND_DELETE,
                    )
                    async with queue_receiver:
                        while True:
                            messages = await queue_receiver.receive_messages(max_message_count=1, max_wait_time=30)
                            if messages:
                                logging.debug(f"Received: {messages[0]}")
        except Exception:
            logging.exception(f"== Running for {datetime.now()-start}")
            await sleep(10)


if __name__ == "__main__":
    loop = get_event_loop()
    loop.run_until_complete(receive_forever())

Please find attached an reduced output log.

reduced.log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

Labels

ClientThis issue points to a problem in the data-plane of the library.Service BusbugThis issue requires a change to an existing behavior in the product in order to be resolved.customer-reportedIssues that are reported by GitHub users external to the Azure organization.needs-author-feedbackWorkflow: More information is needed from author to address the issue.no-recent-activityThere has been no recent activity on this issue.questionThe issue doesn't require a change to the product in order to be resolved. Most issues start as that

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions