fix: use find to infer fields COMPASS-4154 #232
Conversation
mcasimir
changed the title
| @@ -164,7 +165,6 @@ | |||
| "mime-types": "^2.1.24", | |||
| "mongodb-extjson": "^4.0.0-rc1", | |||
| "mongodb-query-parser": "^2.1.2", | |||
| "mongodb-schema": "^8.2.5", | |||
| const docs = await find(ns, filter || {}, { | ||
| limit: sampleSize || DEFAULT_SAMPLE_SIZE, | ||
| ...driverOptions | ||
| }); |
There was a problem hiding this comment.
Just for my own understanding, this will mean that if documents that are added to a collection after the first sampleSize documents will not affect the result here, and that we are okay with that?
There was a problem hiding this comment.
Yep, that is a great point.. I don't think we should be. Initially it was based on 1 document find, then on a random sample of 50 docs, which is a bit better but extremely slow on a big collection, causing the field selection dialog to be blank for a while without any indication, that's why we switched it to a 50 docs find.
The only accurate way to get all the fields is a full collection scan, otherwise we need to change approach: have a 'use all the fields option', relay on users building their own projections without the fields selection, let people decide how many documents to scan to get the fields or at least indicate fields are being loaded...
|
Did this cause https://jira.mongodb.org/browse/COMPASS-4593? |
This is a pre-requisite to COMPASS-4154, since that contains a breaking change for import/export.
Rather than adapting this code to the new
dataService.samplehere we just switch back to a plainfind, since in practice the accuracy is similar but the performance is better compared to a$sampleaggregation.