Dataflow Templates 2024-11-12-00_RC00

Release Week of 2024-11-12

Note: This release is being rolled out. It may not be in your region yet.

Improvements

• [spanner] Handle extra columns present on Spanner (#1995)
• [neo4j] treat empty strings as null values from csv imports (#1996)
• [spanner] Add TOKENLIST spanner type (#1997)

Bug Fixes

N/A

Contributors

• Emrehzl94
• thiagotnunes
• Deep1998

Dataflow Templates 2024-11-06-01_RC00

What's Changed

Improvements

• added level wise table processing by @bharadwaj-aditya in #1983
• Add new metrics in the live template by @shreyakhajanchi in #1913
• test: test reserved keywords in tables by @thiagotnunes in #1991

Bug fixes

• handling for empty tables with bigint keys by @bharadwaj-aditya in #1980
• Escaping column names to handle reserved keywords by @VardhanThigle in #1979
• Fix for reserved words in schema: Reverse replication by @shreyakhajanchi in #1978
• Enabling ANSI Quotes for MYSQL by @VardhanThigle in #1987
• corrected dlq shard id handling by @bharadwaj-aditya in #1988

Full Changelog: 2024-10-29-00_RC00...2024-11-06-01_RC00

Dataflow Templates 2024-10-29-00_RC00

Release Week of 2024-10-29

Improvements

• Added the queryTempDataset option for the BQ read options (#1958)
• Added namespace support for PostgreSQL databases (#1925)
• Terraform template for Reverse replication for sharded MySQL (#1946)

Bug Fixes

N/A

Contributors

• damccorm
• darshan-sj
• fbiville
• liferoad
• manitgupta
• thiagotnunes

Dataflow Templates 2024-10-24-00_RC00

Release Week of 2024-10-28

Improvements

• Added secret manager support for sqlserver-to-bigquery template (#1947)
• Spanner: Added support for vector indexes (#1944)
• Bump Beam version to 2.60.0 (#1948)
• Bump version of avro to avoid CVE-2024-47561 (#1917)
• CassandraToBigtable classic template feature: Cassandra Writetime replication (#1653)
• feat: add PostgreSQL string pkeys support in SourceDB to Spanner (#1905)

Bug Fixes

• bug(bigtable): Handle default values (#1940)

Contributors

• Astha Mohta
• Danny McCormick
• Deep1998
• George Ma
• Jeff Kinard
• Ron Gal
• Thiago Nunes
• Vardhan Vinay Thigle
• Yi Hu
• aksharauke
• atask-g
• darshan-sj
• fayezinislam
• shreyakhajanchi

Dataflow Templates 2024-10-23-00_RC00

Release week of 2024-10-21

Dataflow Templates 2024-10-16-00_RC00

Release week of 2024-10-14

Dataflow Templates 2024-10-15-00_RC00

Release Week of 10/15/2024

Improvements

• Adding JSON enum to userOption for MongoDB templates (#1910)
• Add warnings for excess tables (#1900)
• Add several types to PGAdapter (#1887)
• Reducing MySQL SchemaDiscovery error timeout (#1899)
• Update live terraform templates to support end to end migration (#1889)

Bug Fixes

• Fixed extra table in session file (#1928)
• Fixing wrong timestamp logical type conversion for datastream to bigquery (#1922)

Contributors

• mrtkshi
• Deep1998
• VardhanThigle
• thiagotnunes
• bharadwaj-aditya
• yair-harel

Dataflow Templates 2024-10-14-00_RC00

Release week of 2024-10-14

Dataflow Templates 2024-10-11-00_RC00

Release week of 2024-10-07

Dataflow Templates 2024-10-01-00_RC00

Release week of 2024-09-30