[FLINK-37453][tests] Add state restoration Pipeline E2e cases #3965
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
yuxiqian
force-pushed
the
FLINK-37453-V2
branch
from
46086fb to
357c2f3
Compare
yuxiqian
force-pushed
the
FLINK-37453-V2
branch
from
357c2f3 to
6f2f3ae
Compare
Member
Author
|
CI passed in my forked repo, mark it ready for review. This PR is aimed at migration framework with minimal changes for reviewability. More E2e test cases could be added later to cover state restoring related bugs like FLINK-37537 and FLINK-37538. |
yuxiqian
changed the title
yuxiqian
force-pushed
the
FLINK-37453-V2
branch
from
9182b7f to
c98c474
Compare
yuxiqian
force-pushed
the
FLINK-37453-V2
branch
from
c98c474 to
c3a08da
Compare
# Conflicts: # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToDorisE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MySqlToPaimonE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/MysqlToKafkaE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/OceanBaseE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolveE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/SchemaEvolvingTransformE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/TransformE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/utils/PipelineTestEnvironment.java # Conflicts: # .github/workflows/flink_cdc_ci_nightly.yml # Conflicts: # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/RouteE2eITCase.java # flink-cdc-e2e-tests/flink-cdc-pipeline-e2e-tests/src/test/java/org/apache/flink/cdc/pipeline/tests/UdfE2eITCase.java
yuxiqian
changed the title
yuxiqian
force-pushed
the
FLINK-37453-V2
branch
from
c3a08da to
2190c98
Compare
yuxiqian
force-pushed
the
FLINK-37453-V2
branch
from
2190c98 to
ab0bf9b
Compare
yuxiqian
force-pushed
the
FLINK-37453-V2
branch
from
255407a to
5ca4632
Compare
leonardBang
approved these changes
Apr 25, 2025
Contributor
leonardBang
left a comment
leonardBang
left a comment
There was a problem hiding this comment.
Thanks @yuxiqian for the contribution, as this is a test improvement, +1 to merge once CI passed
linjianchang
pushed a commit
to linjianchang/flink-cdc
that referenced
this pull request
May 16, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This closes FLINK-37453.
CDC pipeline migration tests were introduced in #3447, originally implemented in Ruby scripting language.
However, as most testing logic was written as script file, it is hard to debug or investigate what's going wrong when migration test fails with no much contextual information or stacktrace available. Existing test cases could not be easily extended, too.
This PR tries to extend existing Pipeline E2e test cases to cover such migration tests by:
PipelineTestEnvironmentand remove duplicate code fragments from sub-classes