Resource Processor Not OK

[stuartleeks]

Describe the bug

We have seen errors in the make db-migrate deployment step with the output from the API healthcheck showing the Resource Processor being unhealthy:

{"services":[{"service":"Cosmos DB","status":"OK","message":""},{"service":"Service Bus","status":"OK","message":""},{"service":"Resource Processor","status":"Not OK","message":"Unspecified error"}]}

There is no additional information in the App Insights logs to help diagnose.

Steps to reproduce

This issue has surfaced in environments where we are making performing a high number of builds/deployments.
Running curl <API>/api/health in a bash loop will trigger the issue, and then it will be observed in a deployment to that environment

[stuartleeks]

After adding some additional logging code, I now see this in my environment.

Content: "{\r\n \"error\": {\r\n \"details\": [\r\n {\r\n \"code\": \"TooManyRequests\",\r\n \"message\": \"{\\\"operationGroup\\\":\\\"HighCostGetVMScaleSet30Min\\\",\\\"startTime\\\":\\\"2022-10-24T10:42:36.0142777+00:00\\\",\\\"endTime\\\":\\\"2022-10-24T10:45:00+00:00\\\",\\\"allowedRequestCount\\\":937,\\\"measuredRequestCount\\\":2838}\",\r\n \"target\": \"HighCostGetVMScaleSet30Min\"\r\n }\r\n ],\r\n \"innererror\": {\r\n \"internalErrorCode\": \"TooManyRequestsReceived\"\r\n },\r\n \"code\": \"OperationNotAllowed\",\r\n \"message\": \"The server rejected the request because too many requests have been received for this subscription.\"\r\n }\r\n}"

It appears that there is rate-limiting for the VMSS endpoints. For each call to the healthcheck endpoint we make 2 calls to the VMSS (list and get instance view). Based on the error message, it appears that we can make ~900 calls per 30 minute window.