Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use batch event submission #48

Closed
ePaul opened this issue Jun 21, 2017 · 4 comments
Closed

Use batch event submission #48

ePaul opened this issue Jun 21, 2017 · 4 comments

Comments

@ePaul
Copy link
Member

ePaul commented Jun 21, 2017

Currently, the event submitter does a separate HTTP request for each single event in the queue.
When submitting many events (e.g. for snapshot creation), this has some overhead (network roundtrips, HTTP headers, ...) and slows down the submission.

Instead, we should use the batching-feature of Nakadi (and Fahrschein) to submit several events for the same event type in a batch. Care must be taken to delete only the successful ones from the eventlog table.

@BGehrels BGehrels self-assigned this Jun 29, 2017
@BGehrels
Copy link
Contributor

The current Nakadi Producer API does not define a Batch size limit but it "is planned to be released this quarter" (April 21, 2017). So i guess we should put this ticket on hold until the limit is defined.

@ePaul
Copy link
Member Author

ePaul commented Feb 8, 2018

According to the Nakadi project, they don't have a fixed limit. Batches of up to 50 KB are included in the SLO measuring (of Zalando's internal Nakadi-as-a-service), so that might be a useful guideline.

@BGehrels
Copy link
Contributor

BGehrels commented Feb 5, 2019

Current docs say:

What is the maximum batch size Nakadi could send out? Nakadi maximum batch size is 50 MB of uncompressed data.

BGehrels added a commit that referenced this issue Feb 6, 2019
BGehrels added a commit that referenced this issue Feb 6, 2019
BGehrels added a commit that referenced this issue Feb 6, 2019
ePaul pushed a commit that referenced this issue Feb 6, 2019
ePaul pushed a commit that referenced this issue Feb 6, 2019
ePaul pushed a commit that referenced this issue Feb 6, 2019
ePaul added a commit that referenced this issue Feb 6, 2019
@ePaul ePaul mentioned this issue Feb 6, 2019
ePaul added a commit that referenced this issue Feb 7, 2019
@BGehrels
Copy link
Contributor

Fixed with #113 and experiences are good so far

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants