console.log failure while working with worker threads.

[Mitschiro]

• Version: 12.4.0
• Platform: Docker which runs on mac (Darwin Kernel Version 18.7.0, xnu-4903.278.12~1/RELEASE_X86_64 x86_64)
• Subsystem:

Hello everyone.

Thank you for your hard work on NodeJS.

Now, i developed a small service, which takes 4 csv files, parses them, maps them together and imports them into elasticsearch.

Each file is being parsed on a different thread.

The parsed content of one of the files is being send via an Event to a different file, this file spawns for each set of data, a new thread that will import that set into ES.

In parallel on the main thread, i send the content of one of the files in chunks via an Event with the contents of the remaining 2 files, to a different script again.

Which will spawn a new thread for that chunk of data. That thread will map the given data to the chunk provided, if they match. Send the mapped data back to the main thread, which again will spawn a new thread who will import the mapped data into ES.

The issue i have here is, that once everything is working at the same time, the only console.logs i get are the ones from the main thread. Everything that is being logged on a worker thread, is being lost somewhere, while the main thread is under load.

Note: The actual code is being processed as it should, it is just the console.logs who do not care.

This makes debugging on worker threads really difficult. Maybe i am missing something.

[GrosSacASac]

As a temporary solution, send logs to the main and log them there.

[Mitschiro]

As a temporary solution, send logs to the main and log them there.

Thanks for answering.
Yes that's what i intended to do for the QA version.

[addaleax]

As a temporary solution, send logs to the main and log them there.

Fwiw, this is how console.log(), and more generally process.stdout and process.stderr, in Workers currently work – they send their data for the main thread and wait for that to process it.

The issue happens because when the main thread is busy, then it also won’t process such data coming from the Workers.

[Mitschiro]

As a temporary solution, send logs to the main and log them there.

Fwiw, this is how console.log(), and more generally process.stdout and process.stderr, in Workers currently work – they send their data for the main thread and wait for that to process it.

The issue happens because when the main thread is busy, then it also won’t process such data coming from the Workers.

I can confirm that. Sending the logs to the main thread does nothing while the main thread is busy.
The only thing that worked for me, was off loading more work from the main thread to the worker threads / event based scripts and even that is tricky to do.

[tniessen]

The only thing that worked for me, was off loading more work from the main thread to the worker threads / event based scripts

That sounds like a good idea. I usually try not to block the main thread at all in those scenarios, to make sure it remains responsive.

In theory, we could allow threads to do I/O directly, but it will be tricky and might lead to unwanted side effects.

[devsnek]

how about new Worker(x, { synchronizedStdio: false })?

[addaleax]

@devsnek What would that do and how would it work?

[devsnek]

@addaleax instead of sending internal messages to the main thread, it would just directly write to stdout/err. messages might be garbled on occasion, but sometimes that isn't a problem.

[addaleax]

@devsnek Libuv won’t let you create multiple handles for the same fd, plus writing synchronously would be problematic for the same reasons for which it is on the main thread (and fs.writeSync() for writing data fails on Windows for non-ASCII characters).

[devsnek]

hmm, aren't they separate event loops?

[addaleax]

They are separete event loops – see #30507 for example crashes when sharing fds between threads.

[devsnek]

but it isn't inherently impossible... you just have to make sure you aren't renumbering, closing, etc.