Skip to content

Make function pinning asynchronous #21

@vsreekanti

Description

@vsreekanti

Right now, the management server synchronously waits for functions to pin, which can cause huge backlogs of status updates. This is bad because -- when the function takes a long time to pin -- on the next pin round, it will think there was a huge influx of thousands of requests, and it will try to add some absurd number of replicas.

Two changes need to be made:

  1. Status updates should be eagerly pulled of the queue (i.e., dequeue everything that's there, not one per loop) because otherwise they get starved after long pin operations.
  2. Pins should be made asynchronous. In the interim, there should be a set of sequestered executors that are waiting for pin responses. Those executors should not be used for other pin operations. If the pin succeeds, put them into function_locations. If it fails, put it back into the pool of available executors.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions