Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Module Name] Dinky1.1 adapts to the pipeline function of FlinkCDC3.2 #3792

Open
2 of 3 tasks
huaxiaowei opened this issue Sep 9, 2024 · 1 comment
Open
2 of 3 tasks
Assignees
Labels
New Feature New feature Waiting for reply Waiting for reply

Comments

@huaxiaowei
Copy link

Search before asking

  • I had searched in the issues and found no similar feature requirement.

Description

Dinky-1.1使用FlinkCDC-3.2的pipeline功能报错。原因是3.2相比与3.1的RouteDef类修改了构造方法,多了个replace-symbol参数。
异常堆栈信息如下
Caused by: java.lang.NoSuchMethodError: 'void org.apache.flink.cdc.composer.definition.RouteDef.(java.lang.String, java.lang.String, java.lang.String)'
at org.dinky.trans.pipeline.YamlTextPipelineDefinitionParser.toRouteDef(YamlTextPipelineDefinitionParser.java:146) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.trans.pipeline.YamlTextPipelineDefinitionParser.lambda$null$2(YamlTextPipelineDefinitionParser.java:93) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at org.dinky.trans.pipeline.YamlTextPipelineDefinitionParser.lambda$parse$3(YamlTextPipelineDefinitionParser.java:93) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at java.util.Optional.ifPresent(Optional.java:183) ~[?:?]
at org.dinky.trans.pipeline.YamlTextPipelineDefinitionParser.parse(YamlTextPipelineDefinitionParser.java:93) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.trans.pipeline.FlinkCDCPipelineOperation.execute(FlinkCDCPipelineOperation.java:94) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.interceptor.FlinkInterceptor.build(FlinkInterceptor.java:53) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.executor.Executor.pretreatExecute(Executor.java:186) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.executor.Executor.executeSql(Executor.java:199) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.app.flinksql.Submitter.executeJob(Submitter.java:364) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.app.flinksql.Submitter.submit(Submitter.java:142) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at org.dinky.app.MainApp.main(MainApp.java:53) ~[dinky-app-1.19-1.1.0-jar-with-dependencies.jar:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:108) ~[flink-dist-1.19.1.jar:1.19.1]
at org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) ~[flink-dist-1.19.1.jar:1.19.1]
... 13 more

Use case

适配3.2功能,因为FlinkCDC-3.2修复了同步到Paimon的重复提交问题,原文如下:
Paimon connector has bumped its dependency to an up-to-date version and could benefit from all bug fixes and improvements. Also, there are other improvements including:

Added dynamic bucket mode support for data distribution cases.
Handle specified column positions correctly.
Fixed duplicate commit issue after failover.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@huaxiaowei huaxiaowei added New Feature New feature Waiting for reply Waiting for reply labels Sep 9, 2024
Copy link

github-actions bot commented Sep 9, 2024

Hello @huaxiaowei, this issue is about CDC/CDCSOURCE, so I assign it to @aiwenmo. If you have any questions, you can comment and reply.

你好 @huaxiaowei, 这个 issue 是关于 CDC/CDCSOURCE 的,所以我把它分配给了 @aiwenmo。如有任何问题,可以评论回复。

@github-actions github-actions bot changed the title [Feature][Module Name] Dinky1.1适配FlinkCDC3.2的pipeline功能 [Feature][Module Name] Dinky1.1 adapts to the pipeline function of FlinkCDC3.2 Sep 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Feature New feature Waiting for reply Waiting for reply
Projects
None yet
Development

No branches or pull requests

2 participants