[Bugfix] Source should throw exception when data types between StarRocks and Flink mismatch #127
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this:
Which issues of this PR fixes :
Fixes #126
Problem Summary(Required) :
Source connector will receive data in arrow format from StarRocks' BE, and data will be converted to Flink's format according to the type mapping between StarRocks and Flink. In StarRocksSourceFlinkRows#genFlinkRows, DataTypeUtils.map will be used to map StarRocks's type to Flink's type. It use the arrow's
MinorType
rather than StarRocks' native type. From the row_batch.cpp#convert_to_arrow_type, we can find that StarRocks's nativeVARCHAR
andCHAR
will both be converted to arrow'sStringType
in C++ orMinorType.VARCHAR
in Java. So if arrow returns aMinorType.VARCHAR
, we can't distinguish it's StarRocks'CHAR
orVARCHAR
, andDataTypeUtils.map
will always mapMinorType.VARCHAR
to Flink'sLogicalTypeRoot.VARCHAR
. So if a column whose StarRocks's type isCHAR
and Flink's type isVARCHAR
, source should throw an exception to tell type incompatibility, but now it can work. We need fix it.Checklist: