-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use batch event submission #48
Comments
The current Nakadi Producer API does not define a Batch size limit but it "is planned to be released this quarter" (April 21, 2017). So i guess we should put this ticket on hold until the limit is defined. |
According to the Nakadi project, they don't have a fixed limit. Batches of up to 50 KB are included in the SLO measuring (of Zalando's internal Nakadi-as-a-service), so that might be a useful guideline. |
Current docs say:
|
Fixed with #113 and experiences are good so far |
Currently, the event submitter does a separate HTTP request for each single event in the queue.
When submitting many events (e.g. for snapshot creation), this has some overhead (network roundtrips, HTTP headers, ...) and slows down the submission.
Instead, we should use the batching-feature of Nakadi (and Fahrschein) to submit several events for the same event type in a batch. Care must be taken to delete only the successful ones from the eventlog table.
The text was updated successfully, but these errors were encountered: