Limiting the number of query results incorrectly applied due to batchSize

[mrfelton]

When using the mongo connector, applying a limit to a query does not limit the results properly. Actually, the number of results returned corresponds to the batchSize that is set for Mongo (1000 in my case). Query limits work correctly up to the point where the batchSize is reached.

For example:

curl -s "http://127.0.0.1:3000/api/v1/Events?filter\[limit\]=100" | jq '. | length' # returns 100 results
curl -s "http://127.0.0.1:3000/api/v1/Events?filter\[limit\]=500" | jq '. | length' # returns 500 results
curl -s "http://127.0.0.1:3000/api/v1/Events?filter\[limit\]=1000" | jq '. | length' # returns 1000 results
curl -s "http://127.0.0.1:3000/api/v1/Events?filter\[limit\]=1200" | jq '. | length' # returns 2000 results
curl -s "http://127.0.0.1:3000/api/v1/Events?filter\[limit\]=2500" | jq '. | length' # returns 3000 results

and so on. (I'm using http://stedolan.github.io/jq/ to count the returned results easily).

[mrfelton]

The original patch (explicitly setting batchSize) worked up until the point where the limited result set was upto 7002 results in length. Beyond this, an unpredictable number of results would be returned.

I have refactored the all() method to use each() to iterate through the results rather than using toArray() which seems to be the cause of the issue. Using each(), I get the correctly limited result set in all cases.

[ghislainf]

I have the same issue. @mrfelton your fix works well ! Please merge