-
Notifications
You must be signed in to change notification settings - Fork 34
Open
Description
Right now, the management server synchronously waits for functions to pin, which can cause huge backlogs of status updates. This is bad because -- when the function takes a long time to pin -- on the next pin round, it will think there was a huge influx of thousands of requests, and it will try to add some absurd number of replicas.
Two changes need to be made:
- Status updates should be eagerly pulled of the queue (i.e., dequeue everything that's there, not one per loop) because otherwise they get starved after long pin operations.
- Pins should be made asynchronous. In the interim, there should be a set of sequestered executors that are waiting for pin responses. Those executors should not be used for other pin operations. If the pin succeeds, put them into
function_locations. If it fails, put it back into the pool of available executors.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels