-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor(handlers): remove callback from async lambda handlers - EUBFR-229 #184
refactor(handlers): remove callback from async lambda handlers - EUBFR-229 #184
Conversation
…factor/async-lambda-handlers-EUBFR-229
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@kalinchernev I've tested it and it runs pretty smooth. I got a sense of good performance.
The upload progression seems smoother.
But when testing the code locally i got no error, just a warning.
foo@bar:~$ yarn test
...
Please wait. Server is initializing (parsed files 35000): /@eubfr/storage-signed-uploads: (node:16339) UnhandledPromiseRejectionWarning: Error: BUCKET and REGION environment variables are required!
@eubfr/storage-signed-uploads: (node:16339) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
@eubfr/storage-signed-uploads: (node:16339) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code
....
No errors! 👍
@MrGRA thanks for the review and the catch on the tests. I updated the tests for the new way the handler is defined. |
PR description
This pull request removes all callbacks from lambda handlers, and also other AWS SDK functions where possible. This makes the codebase follow the best practices.
Also error handling is being improved by abstracting the error handler a bit in order to make it more reusable. Common ETL-related utility functions are moved to the existing library module.
I tried to also refactor the streams' pipelines separating the relatively common way of thinking of parsers, transformers and uploaders, however it turned out that AWS are caching variables which are out of the scope of the handlers. Thus, for instance separating the parser in a node module, and having for instance cordis with a few files to be handled subsequently in short period of time, the stream from the first invokation was reused wrongly in the second invokation, yielding "write after end" error. Thus stream-related logic is kept in the handlers for a reason.