Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(handlers): remove callback from async lambda handlers - EUBFR-229 #184

Merged
merged 21 commits into from
Jan 24, 2019

Conversation

kalinchernev
Copy link
Contributor

@kalinchernev kalinchernev commented Jan 11, 2019

PR description

This pull request removes all callbacks from lambda handlers, and also other AWS SDK functions where possible. This makes the codebase follow the best practices.

Also error handling is being improved by abstracting the error handler a bit in order to make it more reusable. Common ETL-related utility functions are moved to the existing library module.

I tried to also refactor the streams' pipelines separating the relatively common way of thinking of parsers, transformers and uploaders, however it turned out that AWS are caching variables which are out of the scope of the handlers. Thus, for instance separating the parser in a node module, and having for instance cordis with a few files to be handled subsequently in short period of time, the stream from the first invokation was reused wrongly in the second invokation, yielding "write after end" error. Thus stream-related logic is kept in the handlers for a reason.

@kalinchernev kalinchernev requested a review from a team January 16, 2019 14:00
@MrGRA MrGRA self-assigned this Jan 22, 2019
@kalinchernev kalinchernev changed the title refactor(handlers): remove callback from async lambda handlers - EUBFR229 refactor(handlers): remove callback from async lambda handlers - EUBFR-229 Jan 23, 2019
Copy link
Collaborator

@MrGRA MrGRA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kalinchernev I've tested it and it runs pretty smooth. I got a sense of good performance.
The upload progression seems smoother.

But when testing the code locally i got no error, just a warning.

foo@bar:~$ yarn test
...
Please wait. Server is initializing (parsed files 35000): /@eubfr/storage-signed-uploads: (node:16339) UnhandledPromiseRejectionWarning: Error: BUCKET and REGION environment variables are required!
@eubfr/storage-signed-uploads: (node:16339) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
@eubfr/storage-signed-uploads: (node:16339) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code
....
No errors! 👍 

@kalinchernev
Copy link
Contributor Author

@MrGRA thanks for the review and the catch on the tests. I updated the tests for the new way the handler is defined.

@kalinchernev kalinchernev merged commit 06d9faa into master Jan 24, 2019
@kalinchernev kalinchernev deleted the refactor/async-lambda-handlers-EUBFR-229 branch January 24, 2019 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants