v0.32.4

What’s Changed

• add input_format_json_infer_incomplete_types_as_strings to default settings (#3378) @heavycrystal
• Add a metric for backfilled TOAST values from CDC store (#3376) @ilidemi
• Sync flow sleeping: change reset time, remove 1hr 20min cap errors (#3368) @Amogh-Bharadwaj
• replace normalize requests channel (#3369) @serprex
• feat(metrics): duration since last flow update (#3374) @iamKunalGupta
• catalog pool: keep connections at 1 aggressively (#3372) @serprex
• quote PartitionKey into sql statement (#3373) @heavycrystal

v0.32.3

What’s Changed

• cleanup RecordMetrics (#3371) @serprex
• DROP TABLE peerdb_stats.cdc_batch_table (#3370) @serprex
• Optimisation: Do not open destination connection for last offset fetch unless Postgres to Postgres (#3367) @Amogh-Bharadwaj
• refactor(aws-creds-cache): set default cache expiry to 1 hour (#3365) @iamKunalGupta

v0.32.2

What's Changed

• follow to memalloc classify by @heavycrystal in #3346
• Mongo codec: test more carefully around the float limit boundary by @ilidemi in #3348
• pg: ignore 58P02 (Duplicate File) by @serprex in #3347
• Replace local activities with plain activities, except status in catalog by @ilidemi in #3350
• mongo: enable ShardedCluster + better code sharing for validation by @jgao54 in #3349
• mongo: move schema logic to shared logic by @jgao54 in #3357
• remove PEERDB_ENABLE_PARALLEL_SYNC_NORMALIZE by @serprex in #3354
• feat: upgrade docker-compose.yml stable image tags by @github-actions[bot] in #3306
• chore(deps): update docker-compose dependencies by @renovate[bot] in #3359
• chore(deps): update github-actions dependencies by @renovate[bot] in #3360
• chore(deps): update dockerfile dependencies by @renovate[bot] in #3363
• 3 max conn, no slot sizing parallelism by @serprex in #3361
• chore: update dependencies by @serprex in #3362
• Create a generic latch.go and make schemas use that by @iskakaushik in #3353

Full Changelog: v0.32.1...v0.32.2

v0.32.1

What's Changed

• Classify replication slot missing by @Amogh-Bharadwaj in #3296
• e2e test for json types should test mysql too by @jgao54 in #3295
• always get status from catalog by @serprex in #3270
• num partitions override by @serprex in #3294
• feat(classifier): tls errors as connectivity by @iamKunalGupta in #3302
• fix(record-slot-sizes): add timeout and record slot information in parallel by @iamKunalGupta in #3304
• refactor(fetched-bytes): metric only looks at mapped tables by @iamKunalGupta in #3291
• Another CDC graph endpoint by @Amogh-Bharadwaj in #3281
• mongo: modify ListCollectionNames and ListCollectionNames by @jgao54 in #3312
• remove tables: use selector by @serprex in #3310
• mongo fixes, adjust log prefix by @heavycrystal in #3313
• [mongo] parallel snapshot for ObjectID key by @heavycrystal in #3293
• mongo: handle inf/-inf/NaN in connector by @jgao54 in #3319
• mongo: set allow_nullable_key = 1 only when custom sort keys are used by @jgao54 in #3303
• fallback to full table partitions for Timescale compressed hypertables by @heavycrystal in #3318
• Add binlog and changestream events timestamp gauge by @Amogh-Bharadwaj in #3314
• mongo: preserve float type from mongo to ch by @jgao54 in #3324
• revert preventing any options in Mongo URIs by @heavycrystal in #3328
• doc: status update in README by @jgao54 in #3329
• fix(deps): update dependency material-symbols to ^0.34.0 by @renovate[bot] in #3332
• mysql: avoid count by using information schema table row count estimate by @serprex in #3334
• Mongo: limit max length of marshaled floats by @ilidemi in #3327
• Introduce log retention gauge by @Amogh-Bharadwaj in #3326
• mysql: ignore zerofill on type by @serprex in #3342
• feat(kafka): add support for custom certificates by @JanPetr in #3321
• classify: still use isClickHouseMvError with QRepSyncError by @serprex in #3344
• clickhouse: PARTITION BY expression by @serprex in #3341
• classify postgres slot allocation issue by @heavycrystal in #3320
• refactor(alerting): remove validation telemetry events by @iamKunalGupta in #3345
• Validate mirror: no need to get source schemas for resync by @Amogh-Bharadwaj in #3325

New Contributors

• @JanPetr made their first contribution in #3321

Full Changelog: v0.32.0...v0.32.1

v0.31.0

What's Changed

This release makes a lot of progress on the MongoDB connector and also adds support for ClickHouse's CoalescingMergeTree, along with some cleanup in the PeeDB UI side.

• mongo: add read preference to mongo config by @jgao54 in #3200
• Timezone cleanup by @serprex in #3202
• feat: upgrade docker-compose.yml stable image tags by @github-actions[bot] in #3207
• [ui] update settings page layout to flex-col by @pratheekrebala in #3214
• [mongo] try to save resumeToken even when idle by @heavycrystal in #3203
• [mongo] fix rootCa handling by @heavycrystal in #3208
• remove trimBoth, don't try parsing raw json in ch normalize by @serprex in #3210
• mongo: make sure events from irrelevant tables are filtered out by @jgao54 in #3196
• mongo: use enum for readPreference instead of string by @jgao54 in #3217
• refactor(classifier): add new pattern for "not found in user directories" by @iamKunalGupta in #3218
• mongo: fix bug in getPartitionForStartAndEnd by @jgao54 in #3222
• further simplify mysql partitioning by @serprex in #3223
• mongo: log a warning for unsupported events by @jgao54 in #3224
• mongo: always return full table partition by @jgao54 in #3227
• PEERDB_ORIGIN_METADATA_AS_DESTINATION_COLUMN by @serprex in #3209
• use bitnami docker images by @serprex in #3229
• Update README.md by @morsapaes in #3230
• Fix SSH input component by @Amogh-Bharadwaj in #3235
• [mongo] SSH tunnelling and metered connection by @heavycrystal in #3231
• fix(classifier): add case of invalid snapshot by @iamKunalGupta in #3238
• feat(classifier): rds binlog invalidation by @iamKunalGupta in #3241
• feat(classifier): ch cannot read from socket as recoverable by @iamKunalGupta in #3240
• mongo: test nested document at limit by @jgao54 in #3242
• refactor custom order by ui by @serprex in #3245
• CoalescingMergeTree by @serprex in #3251
• ch: allow configuring clickhouse destination tables' PARTITION BY by @serprex in #3239
• fix(classifier): normalization errors as mv by @iamKunalGupta in #3243
• mongo: handle validation edge case by @jgao54 in #3257
• remove custom sync by @serprex in #3256
• s3/ch: PEERDB_S3_UUID_PREFIX by @serprex in #3258
• add tables: adding tables to publication should also be part of selector for loop by @serprex in #3254
• update workflow status to FAILED when completing with error by @serprex in #3255
• mongo: handle resume token from table not in changestream by @jgao54 in #3259
• mongo: remove manual computation of bytes ingested by @jgao54 in #3267
• feat: upgrade docker-compose.yml stable image tags by @github-actions[bot] in #3261
• refactor(metrics): lossy conversion warnings by @iamKunalGupta in #3237
• uuid prefix: job must come after uuid by @serprex in #3271
• refactor(classifier): check for pg conn errors earlier by @iamKunalGupta in #3282
• scheduled tasks as one long running activity by @serprex in #3268
• use uuid.NewString by @serprex in #3284
• Add e2e test bucket to dev config by @ilidemi in #3285
• failing to parse LSNs should not be an error by @heavycrystal in #3286
• ch: redact error 117 INCORRECT_DATA by @serprex in #3289
• pg: add e2e test for json types by @jgao54 in #3292

New Contributors

• @pratheekrebala made their first contribution in #3214
• @morsapaes made their first contribution in #3230

Full Changelog: v0.30.7...v0.31.0

v0.30.9

What’s Changed

• update workflow status to FAILED when completing with error (#3255) @serprex
• add tables: adding tables to publication should also be part of selector for loop (#3254) @serprex
• s3/ch: PEERDB_S3_UUID_PREFIX (#3258) @serprex
• remove custom sync (#3256) @serprex
• mongo: handle validation edge case (#3257) @jgao54
• fix(classifier): normalization errors as mv (#3243) @iamKunalGupta
• ch: allow configuring clickhouse destination tables' PARTITION BY (#3239) @serprex
• CoalescingMergeTree (#3251) @serprex
• chore: update dependencies (#3215) @serprex
• chore(deps): update github-actions dependencies (#3249) @renovate[bot]
• chore(deps): update docker-compose dependencies (#3248) @renovate[bot]
• chore(deps): update dockerfile dependencies (#3247) @renovate[bot]
• refactor custom order by ui (#3245) @serprex
• mongo: test nested document at limit (#3242) @jgao54
• feat(classifier): ch cannot read from socket as recoverable (#3240) @iamKunalGupta
• feat(classifier): rds binlog invalidation (#3241) @iamKunalGupta
• fix(classifier): add case of invalid snapshot (#3238) @iamKunalGupta
• [mongo] SSH tunnelling and metered connection (#3231) @heavycrystal
• Fix SSH input component (#3235) @Amogh-Bharadwaj
• feat: upgrade docker-compose.yml stable image tags (#3226) @github-actions[bot]
• Update README.md (#3230) @morsapaes
• use bitnami docker images (#3229) @serprex
• PEERDB_ORIGIN_METADATA_AS_DESTINATION_COLUMN (#3209) @serprex
• mongo: always return full table partition (#3227) @jgao54
• chore: add log for s3 credentials expiry (#3228) @iamKunalGupta
• mongo: log a warning for unsupported events (#3224) @jgao54

v0.30.8

What’s Changed

• further simplify mysql partitioning (#3223) @serprex
• mongo: fix bug in getPartitionForStartAndEnd (#3222) @jgao54
• refactor(classifier): add new pattern for "not found in user directories" (#3218) @iamKunalGupta
• mongo: use enum for readPreference instead of string (#3217) @jgao54
• mongo: make sure events from irrelevant tables are filtered out (#3196) @jgao54
• remove trimBoth, don't try parsing raw json in ch normalize (#3210) @serprex
• [mongo] fix rootCa handling (#3208) @heavycrystal
• [mongo] try to save resumeToken even when idle (#3203) @heavycrystal
• [ui] update settings page layout to flex-col (#3214) @pratheekrebala
• feat: upgrade docker-compose.yml stable image tags (#3207) @github-actions[bot]
• fix(deps): update dependency zod to v4 (#3162) @renovate[bot]
• chore(deps): update github-actions dependencies (#3212) @renovate[bot]
• chore(deps): update dockerfile dependencies (#3211) @renovate[bot]
• Timezone cleanup (#3202) @serprex
• mongo: add read preference to mongo config (#3200) @jgao54

v0.30.7

What's Changed

• mysql: map 00 day/month to 01 by @serprex in #3146
• fix zooPathPrefix typo by @serprex in #3154
• [mysql] move validation to shared for reuse by @heavycrystal in #3150
• Lower s3 chunking default from 33GB to 10GB by @serprex in #3149
• PEERDB_CLICKHOUSE_INITIAL_LOAD_ALLOW_NON_EMPTY_TABLES by @serprex in #3157
• mongo: mongo version check during validation by @jgao54 in #3166
• MySQL: Move binary format fetch outside by @Amogh-Bharadwaj in #3151
• semver upgrades by @serprex in #3165
• TestStringDataTypes: be parallel by @serprex in #3171
• feat: upgrade docker-compose.yml stable image tags by @github-actions[bot] in #3153
• mongo: add validation for (1) storage engine (2) oplog retention by @jgao54 in #3168
• vitess: only allow during initial load only by @heavycrystal in #3172
• mysql: stat activity by @serprex in #3142
• mongo: update config with fields for username/password/tls by @jgao54 in #3178
• feat(classifier): neon quota exceeded as notify_connectivity by @iamKunalGupta in #3182
• allow editing initial load settings by @heavycrystal in #3160
• mysql: report bytes while ingestion still ongoing by @serprex in #3180
• mongo: use same partial reporting mechanism as mysql by @jgao54 in #3187
• Fix null resync by @serprex in #3186
• mongo: ui schema validation by @jgao54 in #3181
• cdc: only insert 1 entry into flows table by @serprex in #3188
• refactor(classifier): ch unknown table with mv issue as NotifyMvOrView by @iamKunalGupta in #3189
• feat: disable fast-fail for other db versions to prevent cancellation of other jobs by @iamKunalGupta in #3192
• Move validateFlavor out of /shared by @Amogh-Bharadwaj in #3191
• PubSub: fix error return on connection active by @Amogh-Bharadwaj in #3193
• Validate tables on ClickHouse: validate in batches of 200 tables by @Amogh-Bharadwaj in #3169
• correctly render readonly fields in edit mirror ui by @serprex in #3194
• feat(internal): add maintenance ui by @iamKunalGupta in #3117
• API endpoint for total rows synced by mirror by @Amogh-Bharadwaj in #3145

Full Changelog: v0.30.6...v0.30.7

v0.30.6

What’s Changed

• MySQL: Fix mixed case for schema changes (#3125) @Amogh-Bharadwaj
• Convert JSONB to JSON in CH (#3143) @dungeon-master-666
• bq: test enum[], fix enum[], fix partition by jsonb (#3144) @serprex
• ch: Distributed Table Engine (#2997) @serprex
• int256: gob.Register (#3138) @serprex
• mongo: allow write to JSON in CH when FF is enabled (#3119) @jgao54
• PEERDB_SNOWFLAKE_AUTO_COMPRESS (#3137) @serprex
• Numeric to int256 (#3134) @serprex
• golangci-lint 2.2.1 (#3135) @serprex
• Apply Nullable to overridden destination types (#3133) @serprex
• mongo: add an e2e test for cross-collection transactions (#3115) @jgao54
• feat: introduce release drafter (#3131) @iamKunalGupta
• feat: upgrade docker-compose.yml stable image tags (#3118) @github-actions[bot]
• mongo: handle large mongo document (#3108) @jgao54
• chore(deps): update github-actions dependencies (#3128) @renovate[bot]
• chore(deps): update dockerfile dependencies (#3127) @renovate[bot]
• chore(deps): update temporalio/auto-setup docker tag to v1.28 (#3129) @renovate[bot]
• mongo: test mongo versions 6.0/7.0/8.0 + authentication (#3102) @jgao54
• [nexus] remove unused catalog functions (#3122) @heavycrystal

v0.30.5

What's Changed

• Truncate and log unbounded numerics that don't fit into destination type by @ilidemi in #3069
• feat: upgrade docker-compose.yml stable image tags by @github-actions in #3063
• remove nio.Pipe by @serprex in #3103
• mongo by @jgao54 in #3092
• mongo: peerdb ui for mongo by @jgao54 in #3054
• [clickhouse] don't repeat already constructed tables by @heavycrystal in #3109
• feat(classifier): invalid snapshot 3 by @iamKunalGupta in #3113
• mysql: fix qrep over tls by @serprex in #3116

Full Changelog: v0.30.4...v0.30.5