[core] Support to make up the sequence number to nanosecond precision #1247

schnappi17 · 2023-05-27T18:02:59Z

Purpose

Taking System.nanotime() as the time source with high precision, make up to the provided sequence.field to nanosecond if it's inTimestamp data type and with the precision of second or millisecond.

Tests

ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnTimestampSecond
ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnTimestampMilliSecond
ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnTimestampMicroSecond
ChangelogWithKeyFileStoreTableTest#testNanosSequenceNumberOnNonTimestampField

JingsongLi

Can we just introduce something like:

sequence.auto-gen.second-micro: the user provides a long with precision to the second, and we give the complement to the micro.
sequence.auto-gen.mills-micro: the user provides a long with precision to the mills, and we give the complement to the micro.

This has the advantage that

it can be applied to all types
a long value can only be saved up to micro, not to nanos

JingsongLi

And please add some ut cases for SequenceGenerator.

JingsongLi · 2023-05-30T09:37:25Z

Or we can introduce a 'sequence.auto-padding' key, valid values are 'second-to-micro' and 'mills-to-micro' and 'none'. Default is none.

schnappi17 · 2023-05-31T18:08:31Z

@JingsongLi Updated, thanks for you suggestions~

JingsongLi · 2023-06-01T03:44:02Z

paimon-core/src/main/java/org/apache/paimon/table/sink/SequenceGenerator.java

@@ -68,6 +73,32 @@ public long generate(InternalRow row) {
        return generator.generate(row, index);
    }

+    public long generateWithPadding(InternalRow row, CoreOptions.SequenceAutoPadding autoPadding) {


I think we can try to finish this in a simpler way:

public long generateWithPadding(InternalRow row, CoreOptions.SequenceAutoPadding autoPadding) { switch (autoPadding) { case SECOND_TO_MICRO: long value = generate(row); // timestamp returns mills int second = fieldType.is(DataTypeFamily.TIMESTAMP) ? value / 1000 : value; return secondToMicro(second); case MILLIS_TO_MICRO: return millsToMicro(generate(row)); default: throw new UnsupportedOperationException( "Unknown sequence padding mode " + autoPadding.name()); } }

JingsongLi · 2023-06-01T03:45:25Z

paimon-core/src/main/java/org/apache/paimon/table/sink/TableWriteImpl.java

@@ -102,6 +103,20 @@ public SinkRecord writeAndReturn(InternalRow row) throws Exception {
        return record;
    }

+    @VisibleForTesting
+    public T writeAndReturnData(InternalRow row) throws Exception {


why not using writeAndReturn and get row from SinkRecord?

JingsongLi · 2023-06-01T03:46:00Z

docs/content/concepts/primary-key-table.md

@@ -260,6 +260,8 @@ there will be some cases that lead to data disorder. At this time, you can use a
 {{< hint info >}}
 When the record is updated or deleted, the `sequence.field` must become larger and cannot remain unchanged. For example,
 you can use [Mysql Binlog operation time](https://ververica.github.io/flink-cdc-connectors/master/content/connectors/mysql-cdc.html#available-metadata) as `sequence.field`.
+If the provided `sequence.field` doesn't meet the precision, like a rough second or millisecond, you can set `sequence.auto-padding` to `second-to-micro` or `millis-to-micro` so that the precision of sequence number will be made up to microsecond by system. 
+Note that only fields of data type `Timestamp`, `Integer` and `BigInt` which indicates a `second` or `millisecond` can be padded by microseconds.


We can apply this padding to al types.

schnappi17 · 2023-06-02T13:00:30Z

@JingsongLi Thanks, your suggestion simplify the implementation for sure, I updated in the latest commit. Why we need the writeAndReturnData is because we can only get the generated sequence number from the origin KeyValue but not SinkRecord. We need to use the generated sequence number to do comparation and make the assertions.

JingsongLi · 2023-06-08T05:28:39Z

@schnappi17 Please ensure test passing first.

JingsongLi · 2023-06-08T05:29:59Z

paimon-core/src/main/java/org/apache/paimon/table/sink/TableWriteImpl.java

+    @VisibleForTesting
+    public T writeAndReturnData(InternalRow row) throws Exception {
+        keyAndBucketExtractor.setRecord(row);
+        SinkRecord record =


you can create a method: toSinkRecord to reuse code.

schnappi17 · 2023-06-14T11:18:23Z

@JingsongLi Updated, and I didn't see any test failed, maybe it's confused by another pr of mine that UT failed, please help to approve the workflows and let's check it, thanks a lot!

lppsuixn · 2023-06-16T07:57:24Z

paimon-core/src/test/java/org/apache/paimon/table/ChangelogWithKeyFileStoreTableTest.java

+            expectedResult =
+                    "1|10|101|1685530987|1685530987123|2023-05-23T11:22:33|2023-05-23T11:22:33.123|2023-05-23T11:22:33.123456|a2";
+        } else {
+            expectedResult =


Based on the situation mentioned in #1050, a2 should always be the final result.
I mean ,the last record should always as final result.

Based on the situation mentioned in #1050, a2 should always be the final result.

@lppsuixn In this situation, if you just set the sequence.field without auto-padding, the last record is the final result as expected.

Currently, when two records have the same sequence number, the latter one is not the final result. #1050 is aimed at resolving this issue.

@lppsuixn In real world, the current implementation is possible because it is difficult to have two data with the same key in one microsecond at the same time.

JingsongLi

+1

hzjhjjyy · 2023-08-09T08:21:34Z

In the current implementation, it's possible for the "padding" of later data's sequence to be smaller than that of the earlier data,resulting in data errors.
for example:
seconds->micros
currentNanoTime1:1062154,766334,200
currentSecondsTime1:1062154
currentNanoTime2:1062155,676231,100
currentSecondsTime2:1062155
sourceMillis:1691482327000
result_sequence:1691482327,766334 and 1691482327,676231(smaller)

Possible solutions?
set an incremental sequence for the sequence.field, incrementing by one for each incoming data?

lppsuixn · 2023-08-09T08:27:13Z

In the current implementation, it's possible for the "padding" of later data's sequence to be smaller than that of the earlier data,resulting in data errors. for example: seconds->micros currentNanoTime1:1062154,766334,200 currentSecondsTime1:1062154 currentNanoTime2:1062155,676231,100 currentSecondsTime2:1062155 sourceMillis:1691482327000 result_sequence:1691482327,766334 and 1691482327,676231(smaller)

Possible solutions? set an incremental sequence for the sequence.field, incrementing by one for each incoming data?

You are right. This PR does not solve the problem.

schnappi17 force-pushed the PAIMON-1050 branch 3 times, most recently from 115f5cb to 38ef728 Compare May 27, 2023 18:06

schnappi17 changed the title ~~Paimon 1050~~ [core] Support to make up the sequence number to nanosecond precision May 28, 2023

JingsongLi reviewed May 29, 2023

View reviewed changes

schnappi17 force-pushed the PAIMON-1050 branch from 38ef728 to 5e572ea Compare May 31, 2023 18:05

JingsongLi reviewed Jun 1, 2023

View reviewed changes

schnappi17 added 4 commits June 2, 2023 20:12

[core] Support to make up the sequence number to nanosecond precision

9c063cc

update

75166cb

resolve comments

31b913f

simplify the implementation

f016e98

schnappi17 force-pushed the PAIMON-1050 branch from 5e572ea to f016e98 Compare June 2, 2023 12:55

JingsongLi reviewed Jun 8, 2023

View reviewed changes

[core] Add private method to reuse code

f350136

[common] Fix option configuration doc bug

978b0d0

lppsuixn reviewed Jun 16, 2023

View reviewed changes

JingsongLi approved these changes Jun 19, 2023

View reviewed changes

JingsongLi merged commit cdf911b into apache:master Jun 19, 2023

hzjhjjyy mentioned this pull request Sep 19, 2023

[Bug] The configurations of second-to-micro and millis-to-micro within sequence.auto-padding does not meet the expected behavior. #2041

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[core] Support to make up the sequence number to nanosecond precision #1247

[core] Support to make up the sequence number to nanosecond precision #1247

schnappi17 commented May 27, 2023

JingsongLi left a comment

JingsongLi left a comment

JingsongLi commented May 30, 2023

schnappi17 commented May 31, 2023

JingsongLi Jun 1, 2023

JingsongLi Jun 1, 2023

JingsongLi Jun 1, 2023

schnappi17 commented Jun 2, 2023

JingsongLi commented Jun 8, 2023

JingsongLi Jun 8, 2023

schnappi17 commented Jun 14, 2023

lppsuixn Jun 16, 2023 •

edited

Loading

schnappi17 Jun 16, 2023

lppsuixn Jun 16, 2023

JingsongLi Jun 19, 2023

JingsongLi left a comment

hzjhjjyy commented Aug 9, 2023

lppsuixn commented Aug 9, 2023

[core] Support to make up the sequence number to nanosecond precision #1247

[core] Support to make up the sequence number to nanosecond precision #1247

Conversation

schnappi17 commented May 27, 2023

Purpose

Tests

JingsongLi left a comment

Choose a reason for hiding this comment

JingsongLi left a comment

Choose a reason for hiding this comment

JingsongLi commented May 30, 2023

schnappi17 commented May 31, 2023

JingsongLi Jun 1, 2023

Choose a reason for hiding this comment

JingsongLi Jun 1, 2023

Choose a reason for hiding this comment

JingsongLi Jun 1, 2023

Choose a reason for hiding this comment

schnappi17 commented Jun 2, 2023

JingsongLi commented Jun 8, 2023

JingsongLi Jun 8, 2023

Choose a reason for hiding this comment

schnappi17 commented Jun 14, 2023

lppsuixn Jun 16, 2023 • edited Loading

Choose a reason for hiding this comment

schnappi17 Jun 16, 2023

Choose a reason for hiding this comment

lppsuixn Jun 16, 2023

Choose a reason for hiding this comment

JingsongLi Jun 19, 2023

Choose a reason for hiding this comment

JingsongLi left a comment

Choose a reason for hiding this comment

hzjhjjyy commented Aug 9, 2023

lppsuixn commented Aug 9, 2023

lppsuixn Jun 16, 2023 •

edited

Loading