Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move image download to image task #769

Merged
merged 1 commit into from
Oct 27, 2019
Merged

Conversation

AWare
Copy link
Contributor

@AWare AWare commented Oct 27, 2019

Having images in the front task output was causing step function to fail due to output being too large.

@markjamesbutler
Copy link
Member

@AWare I'm not super clear on how this works but I assume we might still have problems if there are lots of images on the Front and that causes the this step to fail. Let's catch-up on Monday. It would be good to know exactly what limit was exceeded and how we can prevent that happening in future.

@AWare
Copy link
Contributor Author

AWare commented Oct 27, 2019

Sorry for the lack of detail.

Previously, the list of images was extracted from each front and output. This would then be input into the image downloading step of the step function. This caused the output/input to grow beyond allowed step function limits. This change moves the collate images step into the image downloader task so they are never exposed to the step function. This will stop the output growing with size of edition,

@AWare AWare merged commit 7b088a7 into master Oct 27, 2019
@AWare AWare deleted the aw-fix-archiver-right-now branch October 27, 2019 10:53
@jeanlauliac
Copy link
Contributor

Curious, @AWare how did you detect the problem in production? Alarms from the step function via email?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants