Avoid concurrent infra provisioning requests

[alexluong]

Describe the bug
Outpost is responsible for creating its own mqs based on the user configuration. On startup, every Outpost service will check if the mqs are provisioned, and if not, Outpost will create them. This behavior may cause issues on the very first run when multiple nodes attempt to provision the same mqs.

To Reproduce
Steps to reproduce the behavior:

1. set up a new set of Azure mq credentials
2. run a cluster of Outpost

Expected behavior
The deliverymq and logmq topics and subscriptions are created.

Screenshots

running...
{"level":"info","ts":1749544317.5334902,"caller":"app/app.go:47","msg":"starting outpost","config_path":".env","service":"api"}
{"level":"info","ts":1749544317.7974298,"caller":"app/app.go:197","msg":"no migrations applied","version":3}
failed to declare topic: failed to create topic outpost-delivery: PUT https://management.azure.com/subscriptions/21f864a1-c750-44d6-b40a-bf62a33541aa/resourceGroups/outpost-rg/providers/Microsoft.ServiceBus/namespaces/outpost-servicebus-271b4344/topics/outpost-delivery
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: MessagingGatewayTooManyRequests
--------------------------------------------------------------------------------
{
  "error": {
    "code": "MessagingGatewayTooManyRequests",
    "message": "SubCode=40900. Conflict. You're requesting an operation that isn't allowed in the resource's current state. To know more visit https://aka.ms/sbResourceMgrExceptions.  ExceptionDescription: sb://outpost-servicebus-271b4344.servicebus.windows.net/outpost-delivery. TrackingId:49463282-fa78-475b-8110-355879cf2d3b_G13, SystemTracker:outpost-servicebus-271b4344.servicebus.windows.net:outpost-delivery, Timestamp:2025-06-10T08:32:01"
  }
}
--------------------------------------------------------------------------------

Process Exit with Code: 1

Additional context
N/A

[alexbouchardd]

That's a significant issue, we could use a Redis based lock to know if a process is already creating the dependancies. Alternatively we'd need to have a CI/CD or manual phase to create the resources.

[moltar]

I think this functionality should be disable-able entirely. We use IaC to pre-create everything, so it doesn't even need to do the checking, if envs were set.

The most surprising bit was needing *-dlq SQS queues to be created. I was shocked to see that Outpost actually tried to create infra for me.

[alexluong]

@moltar

I think this functionality should be disable-able entirely.

This is fair and is something we can consider. The thought process was to make the operation as simple as possible. Given these infra should only be used by Outpost, we wanted to "hide" that implementation details away from the end users. I do think the ability to disable it is important indeed.

I was shocked to see that Outpost actually tried to create infra for me.

This is fair and is probably failure from our side from the documentation perspective.

---

For priority purposes, is this a blocker for you currently or are you able to move forward with Outpost already?

[moltar]

Not a blocker. I just pre created the infra and didn't give Outpost permissions to create infra.