You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
CREATE TABLE mytask_print WITH (
'connector' = 'print'
) LIKE mytask_source (EXCLUDING ALL);
insert into mytask_print(id,tasktype ,taskname ) select id,tasktype ,taskname from mytask_source ;
flink Logs输出日志:
2023-02-08 11:13:19,218 WARN io.debezium.relational.history.DatabaseHistoryMetrics [] - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-02-08 11:13:21,414 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - The table schema discovery for binlog split binlog-split success
2023-02-08 11:13:21,455 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed
2023-02-08 11:13:21,460 INFO org.apache.flink.connector.base.source.reader.SourceReaderBase [] - Adding split(s) to reader: [MySqlBinlogSplit{splitId='binlog-split', offset={ts_sec=0, file=binlog.000026, pos=394858166, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, row=0, event=0}, endOffset={ts_sec=0, file=, pos=-9223372036854775808, kind=NON_STOPPING, row=0, event=0}, isSuspended=false}]
2023-02-08 11:13:21,461 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Starting split fetcher 1
2023-02-08 11:13:21,490 WARN io.debezium.connector.mysql.MySqlConnection [] - Database configuration option 'serverTimezone' is set but is obsolete, please use 'connectionTimeZone' instead
2023-02-08 11:13:21,493 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader [] - BinlogSplitReader is created.
2023-02-08 11:13:21,576 WARN io.debezium.relational.history.DatabaseHistoryMetrics [] - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-02-08 11:13:21,578 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - Starting offset is initialized to {ts_sec=0, file=binlog.000026, pos=394858166, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, row=0, event=0}
2023-02-08 11:13:21,584 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - MySQL current GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052975 does contain the GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954 required by the connector.
2023-02-08 11:13:21,644 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - Server has already purged aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-29695267 GTIDs
2023-02-08 11:13:21,658 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:34052955-34052975 known by the server but not processed yet, for replication are available only GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:34052955-34052975
2023-02-08 11:13:21,658 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Started database history recovery
2023-02-08 11:13:21,658 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Finished database history recovery of 0 change(s) in 0 ms
2023-02-08 11:13:21,738 INFO io.debezium.util.Threads [] - Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = binlog-client
2023-02-08 11:13:21,850 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - GTID set purged on server: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-29695267
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Attempting to generate a filtered GTID set
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - GTID set from previous recorded offset: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - GTID set available on server: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052975
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Using first available positions for new GTID channels
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Relevant GTID set available on server: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052975
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Final merged GTID set to use when connecting to MySQL: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Registering binlog reader with GTID set: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Skip 0 events on streaming start
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Skip 0 rows on streaming start
2023-02-08 11:13:21,854 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-02-08 11:13:21,874 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-02-08 11:13:21,959 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Connected to MySQL binlog at 172.16.128.99:3306, starting at MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=binlog.000026, currentBinlogPosition=394858166, currentRowNumber=0, serverId=0, sourceTime=null, threadId=-1, currentQuery=null, tableIds=[], databaseName=null], partition={server=mysql_binlog_source}, snapshotCompleted=false, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, currentGtidSet=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, restartBinlogFilename=binlog.000026, restartBinlogPosition=394858166, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null, incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]]
2023-02-08 11:13:21,960 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-02-08 11:13:21,963 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Waiting for keepalive thread to start
2023-02-08 11:13:22,065 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Keepalive thread is running
2023-02-08 11:14:07,004 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 1: {transaction_id=null, ts_sec=0, file=binlog.000026, pos=394950488, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34053113, row=0, event=0, server_id=1}
2023-02-08 11:24:06,910 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 2: {transaction_id=null, ts_sec=0, file=binlog.000026, pos=396597525, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34055931, row=0, event=0, server_id=1}
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
flink主要配置:
state.backend: rocksdb
state.backend.rocksdb.localdir: /opt/bigdata/data/rocksdb
state.checkpoints.dir: file:///opt/bigdata/data/flink-checkpoints
state.savepoints.dir: file:///opt/bigdata/data/flink-savepoints
flink sql:
SET 'execution.checkpointing.interval' = '600s';
CREATE TABLE mytask_source(
id
int,tasktype
int ,taskname
varchar(255) ,PRIMARY KEY (
id
) NOT ENFORCED) WITH (
'connector' = 'mysql-cdc',
'hostname' = '172.16.128.99',
'port' = '3306',
'username' = 'test',
'password' = 'test',
'server-id'='5408',
'scan.incremental.snapshot.enabled' = 'true',
'database-name' = 'binlogdb',
'table-name' = 'mytask',
'server-time-zone' = 'Asia/Shanghai'
);
CREATE TABLE mytask_print WITH (
'connector' = 'print'
) LIKE mytask_source (EXCLUDING ALL);
insert into mytask_print(
id
,tasktype
,taskname
) selectid
,tasktype
,taskname
from mytask_source ;flink Logs输出日志:
2023-02-08 11:13:19,218 WARN io.debezium.relational.history.DatabaseHistoryMetrics [] - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-02-08 11:13:21,414 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - The table schema discovery for binlog split binlog-split success
2023-02-08 11:13:21,455 INFO io.debezium.jdbc.JdbcConnection [] - Connection gracefully closed
2023-02-08 11:13:21,460 INFO org.apache.flink.connector.base.source.reader.SourceReaderBase [] - Adding split(s) to reader: [MySqlBinlogSplit{splitId='binlog-split', offset={ts_sec=0, file=binlog.000026, pos=394858166, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, row=0, event=0}, endOffset={ts_sec=0, file=, pos=-9223372036854775808, kind=NON_STOPPING, row=0, event=0}, isSuspended=false}]
2023-02-08 11:13:21,461 INFO org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher [] - Starting split fetcher 1
2023-02-08 11:13:21,490 WARN io.debezium.connector.mysql.MySqlConnection [] - Database configuration option 'serverTimezone' is set but is obsolete, please use 'connectionTimeZone' instead
2023-02-08 11:13:21,493 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSplitReader [] - BinlogSplitReader is created.
2023-02-08 11:13:21,576 WARN io.debezium.relational.history.DatabaseHistoryMetrics [] - Unable to register the MBean 'debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source': debezium.mysql:type=connector-metrics,context=schema-history,server=mysql_binlog_source
2023-02-08 11:13:21,578 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - Starting offset is initialized to {ts_sec=0, file=binlog.000026, pos=394858166, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, row=0, event=0}
2023-02-08 11:13:21,584 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - MySQL current GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052975 does contain the GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954 required by the connector.
2023-02-08 11:13:21,644 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - Server has already purged aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-29695267 GTIDs
2023-02-08 11:13:21,658 INFO com.ververica.cdc.connectors.mysql.debezium.task.context.StatefulTaskContext [] - GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:34052955-34052975 known by the server but not processed yet, for replication are available only GTID set aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:34052955-34052975
2023-02-08 11:13:21,658 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Started database history recovery
2023-02-08 11:13:21,658 INFO io.debezium.relational.history.DatabaseHistoryMetrics [] - Finished database history recovery of 0 change(s) in 0 ms
2023-02-08 11:13:21,738 INFO io.debezium.util.Threads [] - Requested thread factory for connector MySqlConnector, id = mysql_binlog_source named = binlog-client
2023-02-08 11:13:21,850 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - GTID set purged on server: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-29695267
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Attempting to generate a filtered GTID set
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - GTID set from previous recorded offset: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - GTID set available on server: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052975
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Using first available positions for new GTID channels
2023-02-08 11:13:21,851 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Relevant GTID set available on server: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052975
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Final merged GTID set to use when connecting to MySQL: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Registering binlog reader with GTID set: aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Skip 0 events on streaming start
2023-02-08 11:13:21,853 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Skip 0 rows on streaming start
2023-02-08 11:13:21,854 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-02-08 11:13:21,874 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-02-08 11:13:21,959 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Connected to MySQL binlog at 172.16.128.99:3306, starting at MySqlOffsetContext [sourceInfoSchema=Schema{io.debezium.connector.mysql.Source:STRUCT}, sourceInfo=SourceInfo [currentGtid=null, currentBinlogFilename=binlog.000026, currentBinlogPosition=394858166, currentRowNumber=0, serverId=0, sourceTime=null, threadId=-1, currentQuery=null, tableIds=[], databaseName=null], partition={server=mysql_binlog_source}, snapshotCompleted=false, transactionContext=TransactionContext [currentTransactionId=null, perTableEventCount={}, totalEventCount=0], restartGtidSet=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, currentGtidSet=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34052954, restartBinlogFilename=binlog.000026, restartBinlogPosition=394858166, restartRowsToSkip=0, restartEventsToSkip=0, currentEventLengthInBytes=0, inTransaction=false, transactionId=null, incrementalSnapshotContext =IncrementalSnapshotContext [windowOpened=false, chunkEndPosition=null, dataCollectionsToSnapshot=[], lastEventKeySent=null, maximumKey=null]]
2023-02-08 11:13:21,960 INFO io.debezium.util.Threads [] - Creating thread debezium-mysqlconnector-mysql_binlog_source-binlog-client
2023-02-08 11:13:21,963 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Waiting for keepalive thread to start
2023-02-08 11:13:22,065 INFO io.debezium.connector.mysql.MySqlStreamingChangeEventSource [] - Keepalive thread is running
2023-02-08 11:14:07,004 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 1: {transaction_id=null, ts_sec=0, file=binlog.000026, pos=394950488, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34053113, row=0, event=0, server_id=1}
2023-02-08 11:24:06,910 INFO com.ververica.cdc.connectors.mysql.source.reader.MySqlSourceReader [] - Binlog offset on checkpoint 2: {transaction_id=null, ts_sec=0, file=binlog.000026, pos=396597525, kind=SPECIFIC, gtids=aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa:1-34055931, row=0, event=0, server_id=1}
Beta Was this translation helpful? Give feedback.
All reactions