Skip to content

Set the number of events per flush to 1,000 to reduce memory footprint #596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 22, 2023

Conversation

zihejia
Copy link
Contributor

@zihejia zihejia commented Mar 22, 2023

This will greatly improve the performance and reduce crashes. Thanks @thibauddavid, #594

@zihejia zihejia requested a review from jaredmixpanel March 22, 2023 17:02
@zihejia zihejia changed the title Add limit to number of events for each flush to 1000 Add 1000 limit to number of events for each flush Mar 22, 2023
let peopleQueue = self.mixpanelPersistence.loadEntitiesInBatch(type: self.persistenceTypeFromFlushType(.people))
let groupsQueue = self.mixpanelPersistence.loadEntitiesInBatch(type: self.persistenceTypeFromFlushType(.groups))
let peopleQueue = self.mixpanelPersistence.loadEntitiesInBatch(type: self.persistenceTypeFromFlushType(.people), batchSize: APIConstants.flushSize)
let groupsQueue = self.mixpanelPersistence.loadEntitiesInBatch(type: self.persistenceTypeFromFlushType(.groups), batchSize: APIConstants.flushSize)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if there are more than 1000 events? should we automatically queue the next flush call? we don't want to knowingly leave events un-flushed while we have the opportunity to flush them. it might be more than 5 days until the next flush and the events will be rejected.

@zihejia zihejia changed the title Add 1000 limit to number of events for each flush The number of events for each flush limit to 1000 for reducing the memory footprint Mar 22, 2023
@zihejia zihejia changed the title The number of events for each flush limit to 1000 for reducing the memory footprint set the number of events for each flush to 1000 for reducing the memory footprint Mar 22, 2023
@zihejia zihejia changed the title set the number of events for each flush to 1000 for reducing the memory footprint Set the number of events per flush to 1,000 to reduce memory footprint Mar 22, 2023
@zihejia zihejia force-pushed the zihe-fix-memory-issue branch from 6e483be to 442606e Compare March 22, 2023 22:49
Copy link
Contributor

@jaredmixpanel jaredmixpanel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zihejia zihejia merged commit cedc74a into master Mar 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants