ServiceBus trigger with managed identity doesn't wake up consumption plan function apps when Connection isn't explicitly defined

[Archomeda]

This is for in-process v4 function apps, I'm unsure whether this also applies to the isolated variant.

When a Service Bus trigger is defined with a connection via managed identity, the function app doesn't get woken up by Azure when the Connection property is not defined in the trigger attribute, despite the trigger working fine when the function app is active.

Investigative information

• Timestamp: 2022-03-22T02:58:56.0115521Z
• Invocation ID: 6309186b-c3f9-43f1-abeb-6f6385cd7c10 (one of many)
• Region: West Europe

Repro steps

1. Create a function app with a service bus trigger via managed identity without specifying the Connection property
2. Publish to Azure
3. Wait for the function app to go inactive
4. Post a message to the service bus queue

Expected behavior

The function app should get woken up and the service bus triggered function should get triggered.
As far as I know, the Connection property was never mandatory for triggers that use connection strings, so this feels like an invisible breaking change when moving to managed identity based connections.

Actual behavior

The function app does not get woken up, and the service bus triggered function is not triggered.

Known workarounds

Define the Connection property that points to the name of the configuration (and gets resolved to <ConnectionName>__fullyQualifiedNamespace).

Related information

I've noticed that the generated function.json only includes the connection field if the Connection property is defined. This makes me believe that the default value "ServiceBus" is only considered in the functions runtime, and not in the Azure scale controller for function apps.

Source

[FunctionName("ServiceBusTest")]
public Task Run([ServiceBusTrigger("queue-name")] QueueMessage message)
{
    // This doesn't get triggered if the function app is inactive
}

The second trigger generates a function.json file with the connection property, because it's now explicitly defined:

[FunctionName("ServiceBusTest2")]
public Task Run([ServiceBusTrigger("queue-name", Connection = "ServiceBus")] QueueMessage message)
{
    // This does get triggered if the function app is inactive
}

[v-bbalaiagar]

Hi @Archomeda , Thank you for your feedback! We will check for the possibilities internally and update you with the findings.

[Xodust]

I am experiencing the same issue, but the work around does not work for me. Any word on a resolution to the issue?

[fabiocav]

@TsuyoshiUshio would you be able to validate/investigate this use case?

@mattchenderson for awareness/comments

[fabiocav]

Reached out to the Scale Controller team for investigation.

[TsuyoshiUshio]

Unfortunately, Scale Controller Default with SeviceBus doesn't support. However, we support AzureWebJobsServiceBus. However, the V5 extension looks support ServiceBus for default value. https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/servicebus/Microsoft.Azure.WebJobs.Extensions.ServiceBus/src/Config/ServiceBusClientFactory.cs#L61 We are working on Scale Controller improvement; however, this fix won't come soon. So that, as mitigation, could you please not use default configuration for now.

[boylec]

Sorry I'm not following with this response from @TsuyoshiUshio

To be clear, how are consumers of Azure Service Bus that want to use Managed Identity supposed to bind their functions?

Are you saying that we can't use ServiceBus__fullyQualifiedNamespace as an app setting and expect functions that use the ServiceBusTrigger to work as designed - instead we must use a different prefix than "ServiceBus"?

Also I think @TsuyoshiUshio's response may be addressing a separate issue having to do with configuration.

I am using "ServiceBusConnection__fullyQualifiedNamespace" and my function does read from service bus queues at least initially.

But once my function idles and JobHost goes to sleep (because I'm on Consumption plan) the next message in the queue does not wake the JobHost back up.

This is not related to the name of the app setting.

When I was opening an Azure Support ticket I saw a suggestion in the "recommendations" step before actually creating the ticket that once a JobHost falls asleep (under consumption plan), it won't wake again until an HTTP request is received from the ScaleController.

Seems this may not be happening in the case of ServiceBusTrigger functions? (No HTTP request from the ScaleController to wake the JobHost back up?)

[boylec]

I believe I have resolved my issue. Details below.

The scale controller (which us end users have no control over from what I can tell) is responsible for waking the JobHost up to consume messages when a service bus queue has any messages.

1. I've added the Azure Service Bus Data Owner role (along w/ Azure Service Bus Data Reader) to my function app's managed identity (which it didn't have before). I am hypothesizing that the scale controller component runs under this managed identity and needs visibility into the queue length to determine whether to spin the function app up or not. Queue length cannot be looked at without the Azure Service Bus Data Owner role. Can anyone confirm that this is how the scale controller works? I'm not 100% on if this is part of what resolved my situation but I did add this role to my function app.
2. This one I'm sure about. My binding for the queue name on my ServiceBusTrigger uses a hierarchical config setting.

public async Task Run([ServiceBusTrigger("%ServiceBus:QueueName:Email%", Connection = "ServiceBusConnection")] string queueMessage)

but my app settings use different hierarchical separators:

ServiceBus__QueueName__Email (no colons)

The scale controller was trying to resolve the queue name by scanning the ServiceBusTrigger attribute in my code (via assembly scanning I'm guessing) and seeing "ServiceBus:QueueName:Email" in that binding, and then trying to resolve "ServiceBus:QueueName:Email" in my app settings but since my app settings use different separators the latter fails.

Long story short: Using consistent hierarchical config separators for both app settings and ServiceBusTrigger binding allows the scale controller to resolve the queue name and start waking up the JobHost again to consume messages.

Super helpful note if you're experiencing issues with your function app waking up for ServiceBusTrigger functions.

Scale controller logging can be enabled using the following as an app setting for the function app. See here for more info on configuring scale controller logging.

SCALE_CONTROLLER_LOGGING_ENABLED=AppInsights:Verbose

After that, you can see these emitted logs by opening log analytics for your app insights instance w/ this query (making sure the scope of your log query is the app insights instance connected to your function app)

traces
| where customDimensions.Category == "ScaleControllerLogs"

This might reveal exceptions happening when the scale controller is trying to parse out your ServiceBusTrigger function binding. If it can't parse your binding it won't know what queue to look at in order to decide whether to wake your JobHost up to consume received messages.

Could be solution for #7762 as well

[hbrotan]

@boylec I'm experiencing the same issue as you:

my function idles and JobHost goes to sleep (because I'm on Consumption plan) the next message in the queue does not wake the JobHost back

I'm trying to use an User Assigned Managed Identity, with connection string: "ServiceBusConnection__fullyQualifiedNamespace"

• @boylec Did you use an User Assigned Managed Identity (UAMI) or System Assigned Managed Identity (SAMI)?

• While using the UAMI, the scale controller gives the following warning:
  [ManagedIdentity] SystemAssigned Managed Identity not found, but the chosen connection method requires one.
  If I add ServiceBusConnection__credential and ServiceBusConnection__clientId to application settings, the ScaleController understands that it needs to use the UAMI.

• I've given the UAMI the role Azure Service Bus Data Owner on the subsciption level.

The scale controller logs:
[ManagedIdentity] Created NamespaceManager with ManagedIdentity
[ManagedIdentity] Created QueueClient with ManagedIdentity
This seems legit.

But the problem still exists -> The messages received when the function is idle will not be processed.

According to the docs (Azure WebJobs Service Bus client library for .NET) this should be straightforward, but there's no mention of scale controller, roles etc.

Can anyone shed a light on this?

[boylec]

@hbrotan I use a system assigned managed identity so not 100% sure how it works with UAMI.

My guess: You most likely need to assign the Azure Service Bus Data Owner and Azure Service Bus Data Reader roles. Have you tried that?

Azure Service Bus Data Owner allows for queue operations, and status reads. Azure Service Bus Data Reader allows the entity to actually consume messages from the queue. Thats my understanding.