Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strange race condition on empy streams #135

Open
pavelhoral opened this issue Jun 13, 2024 · 1 comment
Open

Strange race condition on empy streams #135

pavelhoral opened this issue Jun 13, 2024 · 1 comment

Comments

@pavelhoral
Copy link

pavelhoral commented Jun 13, 2024

What were you expecting to happen?

I have the following Vitest-based test in my project:

import { pipeline } from 'streamx';
import globStream from 'glob-stream';
import toThrough from 'to-through';

describe('globStream', () => {

  it('should concat glob streams', () => new Promise(done => {
    const stream = pipeline(
      globStream('src/*/non-existing.js', { base: 'src', allowEmpty: true }),
      toThrough(globStream('src/foobar/non-existing.js', { base: 'src', allowEmpty: true })),
    );
    stream
      .on('data', () => {})
      .on('end', done);
  }));

});

This test should end without any error. However once in a 10 tries it ends up with timeout (running with vitest). There is a switched order of flush / end events in the second vinyl stream. The issue happens more often if the first glob stream actually matches some files.

I am still trying to find the root cause - whether it is in streamx, to-through or gulp-stream (already spent 8 hours getting to this point so I am quite exhausted).

Terminal output

Successful run:

% npm run -w packages/pipeline test globStream

> <REMOVED-FOR-ISSUE-REPORT>@0.16.1 test
> vitest run globStream


 RUN  v1.6.0 /<REMOVED-FOR-ISSUE-REPORT>

 ✓ src/utils/globStream.spec.ts (1)
   ✓ globStream (1)
     ✓ should concat glob streams

 Test Files  1 passed (1)
      Tests  1 passed (1)
   Start at  23:34:55
   Duration  456ms (transform 79ms, setup 0ms, collect 74ms, tests 11ms, environment 0ms, prepare 151ms)

Failed run:

% npm run -w packages/pipeline test globStream

> @<REMOVED-FOR-ISSUE-REPORT>@0.16.1 test
> vitest run globStream


 RUN  v1.6.0 /<REMOVED-FOR-ISSUE-REPORT>

 ❯ src/utils/globStream.spec.ts (1) 5012ms
   ❯ globStream (1) 5010ms
     × should concat glob streams 5010ms

⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯

 FAIL  src/utils/globStream.spec.ts > globStream > should concat glob streams
Error: Test timed out in 5000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/1]⎯

 Test Files  1 failed (1)
      Tests  1 failed (1)
   Start at  23:34:58
   Duration  5.58s (transform 37ms, setup 0ms, collect 48ms, tests 5.01s, environment 0ms, prepare 164ms)

npm ERR! Lifecycle script `test` failed with error: 
npm ERR! Error: command failed 
npm ERR!   in workspace: @<REMOVED-FOR-ISSUE-REPORT>@0.16.1 
npm ERR!   at location: /<REMOVED-FOR-ISSUE-REPORT>

Please provide the following information:

Library versions (only Gulp 5.0.0 is in my package.json so the rest is transitive):

  • gulp@5.0.0
  • glob-stream@8.0.2
  • streamx@2.18.0
  • to-through@3.0.0

Tested on OSX with NodeJS v20.12.2 and on Kubuntu 24.04 with NodeJS v22.3.0.

@pavelhoral
Copy link
Author

pavelhoral commented Jun 14, 2024

Getting closer to the culprit - seems like the globStream is closed before the pipeline starts flowing. Seems like the issue is that walking is started immediatelly upon stream construction - https://github.com/gulpjs/glob-stream/blob/77826594b4479fe5fefd6a751c28c7f686c24a87/index.js#L286C6-L286C16 and the empty stream can end before the whole pipeline starts flowing.

Got the following sample that IMO illustrates what is happening:

    const readable = new Readable();
    nextTick(() => readable.push(null));
    pipeline(Readable.from([]), toThrough(readable))
      .on('data', () => {})
      .on('end', done); // -> never called

pavelhoral added a commit to pavelhoral/glob-stream that referenced this issue Jun 14, 2024
pavelhoral added a commit to pavelhoral/glob-stream that referenced this issue Jun 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant