MongoDB Full: Use limit parameter as number of total records

... when given.

CHANGES.md

@@ -10,6 +10,7 @@
 - MongoDB Full: Improved initialization of transformation subsystem
 - MongoDB Adapter: Improved performance of when computing collection cardinality
   by using `collection.estimated_document_count()`
+- MongoDB Full: Optionally use `limit` parameter as number of total records
 
 ## 2024/09/19 v0.0.24
 - MongoDB Full: Refactor transformation subsystem to `commons-codec`

cratedb_toolkit/io/mongodb/copy.py

@@ -71,7 +71,11 @@ def start(self):
         Read items from MongoDB table, convert to SQL INSERT statements, and submit to CrateDB.
         """
         logger.info(f"Starting MongoDBFullLoad. source={self.mongodb_uri}, target={self.cratedb_uri}")
-        records_in = self.mongodb_adapter.record_count()
+        limit = self.mongodb_adapter.limit
+        if limit > 0:
+            records_in = limit
+        else:
+            records_in = self.mongodb_adapter.record_count()
         logger.info(f"Source: MongoDB {self.mongodb_adapter.address} count={records_in}")
         with self.cratedb_adapter.engine.connect() as connection, logging_redirect_tqdm():
             if not self.cratedb_adapter.table_exists(self.cratedb_table):

doc/backlog.md

@@ -19,6 +19,7 @@
   mongoimport --uri 'mongodb+srv://MYUSERNAME:SECRETPASSWORD@mycluster-ABCDE.azure.mongodb.net/test?retryWrites=true&w=majority'
   ```
 - MongoDB: Convert dates like `"date": "Sep 18 2015"`, see `testdrive.city_inspections`.
+- Table Loader: Propagate offset/limit to progress bar
 
 ## Iteration +2
 - Address `fix_job_info_table_name`