-
Notifications
You must be signed in to change notification settings - Fork 2
Improve export feature #9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
silvadenisaraujo
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Working with async writes works well and in my perspective solves the memory issue that we have today for large exports, I added comments if you want to go beyond and we could add a mechanism to handle retries and throttling efficiently.
filafb
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Just a comment about handling error in a more clear way, but not a blocker.
filafb
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Just a comment about error handling, but not a blocker at all.
What is the purpose of this pull request?
To improve the export feature, allowing accounts with a huge number of redirects to work (e.g. Americanas)
What problem is this solving?
Currently, Americanas has over 7M registered redirects. However, this plugin stored the redirects in memory, to only save them to a file in the end, causing OOM issues that can kill the OS. This PR changes the process to enqueue the paginated requests and write them to a file concurrently to requesting next pages, which allows a better memory management and avoids OOM issues.
Types of changes
Chores checklist
CHANGELOG.md