Dataflow Templates 2022-04-11-00_RC00

Release Week of 2022-04-11

New Templates

• Spanner Change Streams to GCS (only available in repository)

Improvements

• [Datastream to BigQuery] Use stateful transform to handle Oracle rollbacks
• [Datastream to BigQuery] Use txn id and row id for unique rollback values
• [Dataplex JDBC Ingestion] Added writeDisposition option to allow setting a write strategy if file already exists
• [Spanner Import/Export] Support building change streams DDL from information schema

Bug Fixes

• [Datastream to BigQuery] Change DATETIME handling to account for local timezones in Oracle in replication
• [Dataplex Tiering] Fix TIME field handling

Contributors

• dhercher
• nancyxu123
• olegsa
• pranavbhandari
• 1 internal contributor

Dataflow Templates 2022-04-04-00_RC00

Release Week of 2022-04-04

Improvements

[DataStreamToSQL] Improved logging
[DatastreamToBigQuery] Added datastream logging for json failures

Bug Fixes

[All JDBC Templates] Fix bug that handles JDBC CLOB data converting to String
[Jdbc To BigQuery] Fix bug to get specific Java object from JDBC data type Date and timestamp

Contributors

dhercher
weiwenxu
zhoufek

Dataflow Templates 2022-03-28-00_RC00

Release Week of 2022-03-28

Improvements

Upgraded to Beam 2.37.
[DataStreamToSQL] Added failing SQL statement to error message logged.
[DataStreamToBigQuery] Added an option to add pane info to a windowed filename.
Added first integration test StreamingDataGeneratorIT.

Bug Fixes

Fixed flaky testAssetWithEntityJsonToParquetFailOnExistingFilesE2E test.

Contributors

dhercher
olegsa
pranavbhandari
zhoufek

Dataflow Templates 2022-03-21-00_RC01

Release Week of 2022-03-21

New Templates

Deprecated Datastream to Postgres template (replaced with Datastream to SQL template)

Improvements

[Datastream to SQL] Added support for specifying log level and logging SQL in debug mode
[Spanner to Text] Cleaned up and removed LocalSpannerIO

Bug Fixes

[Pub/Sub to Splunk] Fixed write latency metrics

Contributors

dhercher
pranavbhandari

Dataflow Templates 2022-03-14-00_RC00

Release Week of March 14, 2022 🥧

New Templates

N/A

Improvements

[Pub/Sub to Splunk] Added secret manager support

Bug Fixes

[Datastream Oracle] Map DATE -> DATE instead of DATE -> TIMESTAMP

New Regions

N/A

Contributors

pranavbhandari

Dataflow Templates 2022-03-07-00_RC00

Release Week of March 7, 2022

New Templates

N/A

Improvements

[Pub/Sub to Splunk] Improved metrics

Bug Fixes

N/A

New Regions

N/A

Contributors

pranavbhandari

Dataflow Templates 2022-02-28-00_RC00

Release Week of February 28, 2022

New Templates

N/A

Improvements

[Datastream to SQL] Added support for custom connection strings. This can be used to enable SSL for example, though overriding the sslcert/sslrootcert/sslkey parameters (which expect a local file path) is not yet supported.

Bug Fixes

N/A

New Regions

N/A

Contributors

dhercher
zhoufek

Dataflow Templates 2022-02-21-00_RC00

Release Week of February 21, 2022

New Templates

N/A

Improvements

[All Templates] Upgrade beam version to 2.36
[All Dataplex Templates] Refactor common code

Bug Fixes

[Dataplex: BigQuery to GCS] Dataplex Tiering template: Refactor BigQueryTableToGcsTransform date type fix. This should not change the visible behavior of the transform.

New Regions

N/A

Contributors

olegsa
pranavbhandari
zhoufek

Dataflow Templates 2022-02-07-00_RC01

Release Week of February 7, 2022

This the first release with notes published publicly. We're only including the changes unique to the release, but it contains all released changes up to commit 51f613a, which was pushed on February 7, 2022.

For a list of which templates this covers, please see the Cloud documentation on Google provided templates. Those in the repository that are not listed are currently not part of any release or are only accessible through other services (e.g. Dataplex).

New Templates

N/A

Bug Fixes

[Dataplex: BigQuery to GCS] Dataplex Tiering template: fix for the handling of the DATE fields.
[All Datastore] Fix issue that always set datastoreHintNumWorkers to 500 regardless of user-provided value

Contributors

olegsa
pranavbhandari