QueryStream drops data if event is set late

[seanodell]

I ran into an issue where QueryStream#on("data") misses records if the event is configured late. Below are steps to replicate:

First, populate a test database with a number of records to iterate over (execute in MongoDB shell):

test_db = db.getMongo().getDB("test");

test_db.test.remove();

for ( var index = 0; index < 30000; index++) {
  test_db.test.insert({
    index : index,
    value : Math.floor((Math.random() * 10000) + 1)
  });
}

Then, using QueryStream, iterate over all records:

var host = 'localhost';
var port = 27017;
var db = 'test';
var timeout = 200;

var mongoose = require('mongoose');

var conn = mongoose.createConnection('mongodb://' + host + ':' + port + '/' + db);

var schema = mongoose.Schema({
  index : Number,
  value : Number
});

var model = conn.model('Test', schema, 'test');

var query = model.find({});

var queryStream = query.stream();

var dataCount = 0;

setTimeout(function() {
  queryStream.on('close', function() {
    console.log("Data Count: " + dataCount);
  });

  queryStream.on('error', function(err) {
    console.log("ERROR: " + err);
  });

  queryStream.on('data', function(doc) {
    dataCount += 1;

    if (Math.floor(Math.random() * 100) == 50) {
      queryStream.pause();
      process.nextTick(function() {
        queryStream.resume();
      });
    }
  });
}, timeout);

If you set the timeout value to 0, you will probably get a count of 30000 (the actual number of records). Set it higher and the count drops, eventually down to zero if you set it to 1000ms or more.

It looks like the QueryStream is iterating through records in the background whether or not an event has been set up.

[aheckmann]

this is the node 0.8 style of streams and is working as designed. I'd love to see a PR that supports node 0.10 (pull) streams.

[seanodell]

I know the 0.8 style has back-pressure and other issues, but it's normal for data to start streaming when there's no data handler at all?

I'm not having any luck finding a reference to this anywhere. Hopefully you or someone else can drop a link to something that explains it in detail? I've only been working with node for maybe 9 months, so I can't really argue the point, but it does seem just too odd that data would start flying over the wire and go nowhere because there was no handler set up.

Now that I know this happens, I can modify my code to avoid it completely, but I'd like to understand this better.

Thanks for the response, and thanks again for the work you've put into this project!

[aheckmann]

Yes it's normal. Hence it was changed in node 0.10.

On Thursday, January 30, 2014, seanodell notifications@github.com wrote:

I know the 0.8 style has back-pressure and other issues, but it's normal
for data to start streaming when there's no data handler at all?

I'm not having any luck finding a reference to this anywhere. Hopefully
you or someone else can drop a link to something that explains it in
detail? I've only been working with node for maybe 9 months, so I can't
really argue the point, but it does seem just too odd that data would start
flying over the wire and go nowhere because there was no handler set up.

Now that I know this happens, I can modify my code to avoid it completely,
but I'd like to understand this better.

Thanks for the response, and thanks again for the work you've put into
this project!

Reply to this email directly or view it on GitHubhttps://github.com//issues/1907#issuecomment-33714991
.

Aaron
@aaronheckmann https://twitter.com/#!/aaronheckmann
soundcloud.com/ajhecky
github.com/aheckmann

[helmus]

any update on support for v2 streams ?

[blyork]

I ran a similar issue myself. This is a work around:

var readStream = (new stream.Readable({ objectMode: true })).wrap(Model.find({}).stream());

[vkarpov15]

No update yet. Don't really have a good idea where this fits on the roadmap either - open to suggestions.

[helmus]

Bump !

[kfitzgerald]

I'd consider this problem to be a much more serious issue, especially because it's a real nasty "gotcha".

I ran into this when I discovered that 260,000 docs were dropped when processing a stream batch of 4.5 million docs, using stream.pause and stream.resume as a governor to control memory while processing. There's zero indication that something went wrong.

This should not be a "find out the hard way" sort of issue. At the very least, a well documented warning that pausing the stream may result in data loss, should be immediately warranted.

Note: One difference: I am not late-setting the data event handler, so no records should be missed.