Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move execution of lua transform into a separate thread #2200

Open
ghost opened this issue Apr 1, 2020 · 2 comments
Open

Move execution of lua transform into a separate thread #2200

ghost opened this issue Apr 1, 2020 · 2 comments
Labels
domain: reliability Anything related to Vector's reliability needs: approval Needs review & approval before work can begin. transform: lua Anything `lua` transform related type: tech debt A code change that does not add user value.

Comments

@ghost
Copy link

ghost commented Apr 1, 2020

Currently Lua transform (both version 1 and version 2) execute the user-defined code inside a Tokio thread. However, if the user-defined code hangs, for example because of an infinite loop, it might stop entire topology if there are no other available threads.

It can be fixed by moving the runtime code into a separate thread, and then making this separate thread communicate with the Tokio threads using channels.
@ghost ghost added type: enhancement A value-adding code change that enhances its existing functionality. transform: lua Anything `lua` transform related labels Apr 1, 2020
@ghost ghost mentioned this issue Apr 1, 2020
Merged
@Hoverbear Hoverbear assigned ghost Apr 2, 2020
@ghost ghost added this to the Improve data processing milestone Apr 3, 2020
@ghost
Copy link
Author

ghost commented Apr 10, 2020 •
edited by ghost
Loading

It looks adding asynchronous polling from a different thread might require changes in the topology. I prefer to put this on hold for now.

@binarylogic binarylogic added the have: should We should have this feature, but is not required. It is medium priority. label Apr 17, 2020
@binarylogic binarylogic removed this from the Improve Data Processing milestone Jul 26, 2020
@binarylogic
Copy link
Contributor

@lukesteensen, I've lumped this into the architecture epic given the above comment.

@binarylogic binarylogic added type: tech debt A code change that does not add user value. domain: reliability Anything related to Vector's reliability and removed type: enhancement A value-adding code change that enhances its existing functionality. labels Aug 7, 2020
@binarylogic binarylogic unassigned ghost Aug 7, 2020
@lukesteensen lukesteensen added needs: approval Needs review & approval before work can begin. and removed have: should We should have this feature, but is not required. It is medium priority. labels Sep 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
domain: reliability Anything related to Vector's reliability needs: approval Needs review & approval before work can begin. transform: lua Anything `lua` transform related type: tech debt A code change that does not add user value.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@lukesteensen @binarylogic