Skip to content

Issues in MySQL to BigQuery CDC template #98

@rohansuri

Description

@rohansuri

Thanks for the CDC Dataflow template. I ran the template and faced a couple of issues. If they are valid, I'd be happy to submit a PR.

  • In the properties file, databaseName should actually be named instanceName since it is used as Debezium's database.server.name property.

  • Discrepancy in PubSub Topic name. The Deploying the connector section states:

    A prefix for PubSub topics corresponding to each MySQL table. The connector will push table updates to ${PREFIX}${DB_INSTANCE}.${DATABASE}.${TABLE}

    i.e. the topic name is dot separated. Whereas later Setting up PubSub topics section states it is underscore separated.

    Table: my-mysql.cdc_demo.people
    Topic: export_demo_my-mysql_cdc_demo_people

    It happens to be dot separated actually as per the logic in PubSubChangeConsumer.getPubsubTopicName since table name already is dot separated and there's no String.replace being done there to create the topic name underscore separated. So the docs in latter section needs a change.

  • Order of dataset arguments passed to MergeStatementBuildingFn in BigQueryChangeApplier is incorrect. MergeStatementBuildingFn expects changeLogDataset first.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions