v0.0.29

What's Changed

• MongoDB: Added Zyp transformations to the CDC subsystem,
  making it more symmetric to the full-load procedure.
• Query Converter: Added very basic expression converter utility with
  CLI interface
• DynamoDB: Added query expression converter for relocating object
  references, to support query migrations after the breaking change
  with the SQL DDL schema, by v0.0.27.

Full Changelog: v0.0.28...v0.0.29

v0.0.28

What's Changed

• IO: Improved BulkProcessor when running per-record operations by
  also checking rowcount for handling INSERT OK, 0 rows responses
• MongoDB: Fixed BSON decoding of {"$date": 1180690093000} timestamps
  by updating to commons-codec 0.0.21.
• Testcontainers: Don't always pull the OCI image before starting.
  It is unfortunate in disconnected situations.

Details

• Update poethepoet requirement from <0.29 to <0.30 by @dependabot in #283
• Update validate-pyproject requirement from <0.20 to <0.21 by @dependabot in #284
• IO: Improve BulkProcessor when running per-record operations by @amotl in #286
• Testcontainers: Don't always pull the OCI image before starting. by @amotl in #287

Full Changelog: v0.0.27...v0.0.28

v0.0.27

What's Changed

• DynamoDB: Change CrateDB data model to use (pk, data, aux) columns by @amotl in #281
  Attention: This is a breaking change.

Dependencies

• MongoDB: Update to pymongo 4.9 by @amotl in #277
• Update yarl requirement from <1.12 to <1.14 by @dependabot in #279

Full Changelog: v0.0.26...v0.0.27

v0.0.26

What's Changed

• MongoDB: Configure MongoDBCrateDBConverter for updating commons-codec by @amotl in #275
• DynamoDB CDC: Make MODIFY operation also propagate deleted attributes by @amotl in #276

Dependabot

• Update python-bsonjs requirement from <0.5 to <0.6 by @dependabot in #272
• Bump cratedb-sqlparse from 0.0.6 to 0.0.7 by @dependabot in #274

Full Changelog: v0.0.25...v0.0.26

v0.0.25

What's Changed

• Table Loader: Improved conditional handling of "transformation" parameter
• Table Loader: Improved status reporting and error logging in BulkProcessor
• MongoDB: Improve error reporting
• MongoDB Full: Polars' read_ndjson doesn't load MongoDB JSON data well, use fsspec and orjson instead
• MongoDB Full: Improved initialization of transformation subsystem
• MongoDB Adapter: Improved performance of when computing collection cardinality by using collection.estimated_document_count()
• MongoDB Full: Optionally use limit parameter as number of total records
• MongoDB Adapter: Evaluate _id filter field by upcasting to bson.ObjectId, to convey a filter that makes ctk load table process a single document, identified by its OID
• MongoDB Dependencies: Update to commons-codec 0.0.17

Full Changelog: v0.0.24...v0.0.25

v0.0.24

What's Changed

• MongoDB Full: Refactor transformation subsystem to commons-codec
• MongoDB: Update to commons-codec v0.0.16

Details

• AWS DMS: Add first working version by @amotl in #213
• MongoDB Full: Refactor transformation subsystem to commons-codec by @amotl in #269

Full Changelog: v0.0.23...v0.0.24

v0.0.23

What's Changed

• MongoDB: Unlock processing multiple collections, either from server database, or from filesystem directory
• MongoDB: Unlock processing JSON files from HTTP resource, using https+bson://
• MongoDB: Optionally filter server collection using MongoDB query expression
• MongoDB: Improve error handling wrt. bulk operations vs. usability
• DynamoDB CDC: Add ctk load table interface for processing CDC events
• DynamoDB CDC: Accept a few more options for the Kinesis Stream:
  batch-size, create, create-shards, start, seqno, idle-sleep, buffer-time
• DynamoDB Full: Improve error handling wrt. bulk operations vs. usability

Details

• MongoDB: Improve support for reading JSON/BSON files by @amotl in #261
• MongoDB: Improve error handling wrt. bulk operations vs. usability by @amotl in #262
• DynamoDB: Add ctk load table interface for processing CDC events by @amotl in #247
• Update datasets requirement from <3 to <4 by @dependabot in #264
• Update aiobotocore requirement from <2.15 to <2.16 by @dependabot in #265
• Update cattrs requirement from <24 to <25 by @dependabot in #267
• DynamoDB Full: Improve error handling wrt. bulk operations vs. usability by @amotl in #263

Full Changelog: v0.0.22...v0.0.23

v0.0.22

What's Changed

• MongoDB: Rename columns with leading underscores to use double leading underscores
• MongoDB: Add support for UUID types
• MongoDB: Improve reading timestamps in previous BSON formats
• MongoDB: Fix processing empty arrays/lists. By default, assume TEXT as inner type.
• MongoDB: For ctk load table, use "partial" scan for inferring the collection schema,
  based on the first 10,000 documents.
• MongoDB: Skip leaking UNKNOWN fields into SQL DDL.
  This means relevant column definitions will not be included into the SQL DDL.
• MongoDB: Make ctk load table use the data OBJECT(DYNAMIC) mapping strategy.
• MongoDB: Sanitize lists of varying objects
• MongoDB: Add treatment option for applying special treatments to certain items
  on real-world data
• MongoDB: Use pagination on source collection, for creating batches towards CrateDB
• MongoDB: Unlock importing MongoDB Extended JSON files using file+bson://...

Details

• Bump croud from 1.11.1 to 1.12.0 by @dependabot in #256
• Update yarl requirement from <1.10 to <1.12 by @dependabot in #258
• Update fastapi requirement from <0.113 to <0.115 by @dependabot in #259
• MongoDB: More ad hoc fixes for supporting real-world data by @amotl in #255

Full Changelog: v0.0.21...v0.0.22

v0.0.21

What's Changed

• DynamoDB: Add special decoding for varied lists.
  Store them into a separate OBJECT(IGNORED) column in CrateDB.
• DynamoDB: Add pagination support for full-load table loader

Details

• DynamoDB: Add software test for decoding list of objects by @amotl in #246
• DynamoDB: Improve software test about decoding a list with empty string by @amotl in #248
• Update polars requirement from <1.6 to <1.7 by @dependabot in #250
• Update poethepoet requirement from <0.28 to <0.29 by @dependabot in #251
• Revert "CI: Codecov API is currently down" by @amotl in #253
• DynamoDB: Add pagination support for full-load table loader by @amotl in #252

Full Changelog: v0.0.20...v0.0.21

v0.0.20

What's Changed

• DMS/DynamoDB: Fix table name quoting within CDC processor handler by @amotl in #245

Full Changelog: v0.0.19...v0.0.20