Add streaming upload #194
Add streaming upload #194
Conversation
* Body for put/post/patch can be a {:stream, enumerable}
* Stream.transform puts bytes into the socket lazily,
halting at the first error and emitting that error
|
Thanks for the PR. Sorry for the delay. I will give it a try later today and see how we can deal with errors etc. Thank you very much! 👍 |
| _, :error -> {:halt, :error} | ||
| bin, :ok -> {[], :hackney.send_body(ref, bin)} | ||
| _, error -> {[error], :error} | ||
| end) |> Enum.into([]) |
There was a problem hiding this comment.
so if I understood well we always "materialise" the stream to become a list before sending? Then we are not going to "stream" the upload as it will be in memory before sending the request.
There was a problem hiding this comment.
Saying that I'm not sure how we would actually stream the data..
There was a problem hiding this comment.
Using Enum.into here to build a list of failures. On the happy path, it only emits empty list to Stream.transform after sending the body, so we're not materializing the list before sending. On the failure side it emits the error, and then changes the state to :error so the error can get emitted and then the stream halts on the next go round.
This is why I'm not using Stream.run here - we want the failure back in the case that something goes wrong.
There was a problem hiding this comment.
Gotcha! I completely misunderstood your code :P
{:stream, enumerable}halting at the first error and emitting that error
TODO: Not sure what the best way to test a call to
send_bodyfailing while the stream is being transformed, any thoughts here? Right now there's only a test for the non-failure case.