Support for async requests or webhooks

[brian316]

---

🚀 Feature

Is there currently a way to run inference and get the result by polling or a webhook callback?

Motivation

If an inference takes a long time to run lets say 30 min you dont want the connection to drop because you will have to rerun the inference especially on a shaky connection.

Pitch

A webhook callback or polling method can let us run inference as a background task based off a uuid or something the user specifies so that they can come back at a later time for the result. can also help integrations so the inference is not a blocking request.

Alternatives

Cog from replicate implements a webhook

Additional context

[aniketmaurya]

@brian316 this is a great feature! You can already do this with LitServe 😄

Here are the steps:

1. Send the webhook endpoint in request body
2. Use callback and define hooks at on_after_decode_request , on_after_predict and on_after_encode_response.
3. Send the current status of request/response in each hooks/stage.

[brian316]

I've looked at the documentation and saw those handy Hooks, but wouldn't the the initial api request wait until the server returns something? I was thinking of returning something immediately like an id the user can poll for the polling method or for the web hook method return a success message. while still running the background task. I guess something that could potentially work is returning immediately in the predict function and run some thread in the background but that seems hacky to get it to work. having async task is important for my type of tasks