feat: Add import reader for binlog #28910

bigsheeper · 2023-12-01T10:23:54Z

This PR defines the new import reader interfaces and implement a binlog reader for import.

issue: #28521

mergify · 2023-12-04T05:10:20Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

codecov · 2023-12-04T09:46:43Z

Codecov Report

Attention: 177 lines in your changes are missing coverage. Please review.

Comparison is base (99e0f1e) 83.72% compared to head (1bd6e13) 83.66%.
Report is 16 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #28910      +/-   ##
==========================================
- Coverage   83.72%   83.66%   -0.07%     
==========================================
  Files         679      685       +6     
  Lines      108623   109117     +494     
==========================================
+ Hits        90949    91292     +343     
- Misses      14361    14465     +104     
- Partials     3313     3360      +47

Files	Coverage Δ
internal/datacoord/handler.go	`79.85% <100.00%> (+0.29%)`	⬆️
internal/datanode/syncmgr/storage_serializer.go	`89.92% <100.00%> (+0.07%)`	⬆️
internal/datanode/syncmgr/storage_v2_serializer.go	`80.12% <100.00%> (+0.12%)`	⬆️
internal/datanode/writebuffer/l0_write_buffer.go	`70.37% <100.00%> (+2.37%)`	⬆️
internal/datanode/writebuffer/write_buffer.go	`92.07% <100.00%> (-1.33%)`	⬇️
internal/proxy/meta_cache.go	`89.64% <100.00%> (-0.18%)`	⬇️
internal/proxy/task.go	`85.48% <100.00%> (ø)`
internal/proxy/task_delete.go	`85.98% <100.00%> (ø)`
internal/proxy/task_index.go	`75.74% <100.00%> (ø)`
internal/proxy/task_query.go	`79.26% <100.00%> (ø)`
... and 30 more

... and 17 files with indirect coverage changes

mergify · 2023-12-04T09:50:10Z

@bigsheeper ut workflow job failed, comment rerun ut can trigger the job again.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

…2-binlog-reader

mergify · 2023-12-05T15:23:02Z

@bigsheeper ut workflow job failed, comment rerun ut can trigger the job again.

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

mergify · 2023-12-25T13:15:49Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2023-12-25T16:03:17Z

@bigsheeper ut workflow job failed, comment rerun ut can trigger the job again.

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

mergify · 2024-01-02T04:28:35Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

bigsheeper · 2024-01-02T06:09:44Z

/run-cpu-e2e

internal/util/importutilv2/binlog/util.go

internal/util/importutilv2/reader.go

internal/util/importutilv2/binlog/reader.go

internal/util/importutilv2/binlog/column_reader.go

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

bigsheeper · 2024-01-04T09:33:33Z

rerun ut

…2-binlog-reader

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

czs007 · 2024-01-05T03:44:08Z

internal/util/importutilv2/binlog/reader.go

+		if err != nil {
+			return nil, err
+		}
+		fieldData, err := fr.Next(-1)


fr.Next() ?
you can modify this in next pr?

czs007 · 2024-01-05T03:46:21Z

/approve
/lgtm

sre-ci-robot · 2024-01-05T03:46:30Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigsheeper, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Dec 1, 2023

sre-ci-robot requested review from czs007 and yanliang567 December 1, 2023 10:24

bigsheeper mentioned this pull request Dec 1, 2023

[Feature]: Refactor bulkinsert #28521

Closed

12 tasks

mergify bot added dco-passed DCO check passed. kind/feature Issues related to feature request from users labels Dec 1, 2023

bigsheeper added 3 commits December 4, 2023 17:08

feat: Add import reader for binlog

436ac4d

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

updaet

500955e

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

update import reader for binlog

963891f

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

bigsheeper force-pushed the 2312-binlog-reader branch from 292cbd5 to 963891f Compare December 4, 2023 09:10

bigsheeper changed the title ~~feat: Add binlog reader for import~~ feat: Add import reader for binlog Dec 4, 2023

bigsheeper added 11 commits December 5, 2023 10:10

update reader

0fb4c5d

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

dc9f7d4

…2-binlog-reader

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

e6d590e

…2-binlog-reader

resolve import sycle

cdce1ad

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

code format

cffeda2

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

868688f

…2-binlog-reader

update

691a44a

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

f2e4285

…2-binlog-reader

update test

b96c030

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

14eb29a

…2-binlog-reader

update

61998d3

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

sre-ci-robot added size/XL Denotes a PR that changes 500-999 lines. and removed size/L Denotes a PR that changes 100-499 lines. labels Dec 5, 2023

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

c5fb921

…2-binlog-reader

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

9938ef8

…2-binlog-reader

bigsheeper added 2 commits December 25, 2023 20:47

readPKs

32a5496

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

update

ef0e987

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

bigsheeper added 2 commits December 26, 2023 14:27

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

76c896c

…2-binlog-reader

fix conflicts

4e1f23d

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

mergify bot added the ci-passed label Jan 2, 2024

czs007 reviewed Jan 2, 2024

View reviewed changes

internal/util/importutilv2/binlog/util.go Outdated Show resolved Hide resolved

czs007 reviewed Jan 2, 2024

View reviewed changes

internal/util/importutilv2/reader.go Outdated Show resolved Hide resolved

czs007 reviewed Jan 3, 2024

View reviewed changes

internal/util/importutilv2/reader.go Outdated Show resolved Hide resolved

czs007 reviewed Jan 3, 2024

View reviewed changes

internal/util/importutilv2/reader.go Show resolved Hide resolved

czs007 reviewed Jan 3, 2024

View reviewed changes

internal/util/importutilv2/binlog/reader.go Outdated Show resolved Hide resolved

czs007 reviewed Jan 3, 2024

View reviewed changes

internal/util/importutilv2/binlog/reader.go Outdated Show resolved Hide resolved

czs007 reviewed Jan 3, 2024

View reviewed changes

internal/util/importutilv2/binlog/column_reader.go Outdated Show resolved Hide resolved

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

5d3282d

…2-binlog-reader

mergify bot removed the ci-passed label Jan 4, 2024

update after code review

43ed85a

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

bigsheeper added 2 commits January 4, 2024 20:08

Merge branch 'master' of https://github.com/milvus-io/milvus into 231…

26bb258

…2-binlog-reader

remove fieldReader api

1bd6e13

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

czs007 reviewed Jan 5, 2024

View reviewed changes

sre-ci-robot assigned czs007 Jan 5, 2024

sre-ci-robot added the lgtm label Jan 5, 2024

sre-ci-robot added the approved label Jan 5, 2024

czs007 added the ci-passed label Jan 5, 2024

sre-ci-robot merged commit 3561586 into milvus-io:master Jan 5, 2024
13 of 14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add import reader for binlog #28910

feat: Add import reader for binlog #28910

bigsheeper commented Dec 1, 2023

mergify bot commented Dec 4, 2023

codecov bot commented Dec 4, 2023 •

edited

Loading

mergify bot commented Dec 4, 2023

mergify bot commented Dec 5, 2023

mergify bot commented Dec 25, 2023

mergify bot commented Dec 25, 2023

mergify bot commented Jan 2, 2024

bigsheeper commented Jan 2, 2024

bigsheeper commented Jan 4, 2024

czs007 Jan 5, 2024

czs007 commented Jan 5, 2024

sre-ci-robot commented Jan 5, 2024

feat: Add import reader for binlog #28910

feat: Add import reader for binlog #28910

Conversation

bigsheeper commented Dec 1, 2023

mergify bot commented Dec 4, 2023

codecov bot commented Dec 4, 2023 • edited Loading

Codecov Report

mergify bot commented Dec 4, 2023

mergify bot commented Dec 5, 2023

mergify bot commented Dec 25, 2023

mergify bot commented Dec 25, 2023

mergify bot commented Jan 2, 2024

bigsheeper commented Jan 2, 2024

bigsheeper commented Jan 4, 2024

czs007 Jan 5, 2024

Choose a reason for hiding this comment

czs007 commented Jan 5, 2024

sre-ci-robot commented Jan 5, 2024

codecov bot commented Dec 4, 2023 •

edited

Loading