Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add import scheduler and manager #29367

Merged
merged 254 commits into from
Mar 1, 2024
Merged

feat: Add import scheduler and manager #29367

Changes from 1 commit
Commits
Show all changes
254 commits
Select commit Hold shift + click to select a range
500955e
updaet
bigsheeper Dec 1, 2023
963891f
update import reader for binlog
bigsheeper Dec 4, 2023
0fb4c5d
update reader
bigsheeper Dec 5, 2023
dc9f7d4
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 5, 2023
e6d590e
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 5, 2023
cdce1ad
resolve import sycle
bigsheeper Dec 5, 2023
cffeda2
code format
bigsheeper Dec 5, 2023
868688f
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 5, 2023
691a44a
update
bigsheeper Dec 5, 2023
f2e4285
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 5, 2023
b96c030
update test
bigsheeper Dec 5, 2023
14eb29a
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 5, 2023
61998d3
update
bigsheeper Dec 5, 2023
c5fb921
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 5, 2023
9938ef8
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 6, 2023
7e2f1cf
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 8, 2023
bf0edeb
update test
bigsheeper Dec 11, 2023
3c0a72b
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 13, 2023
20e19f0
update
bigsheeper Dec 13, 2023
4dd1bd9
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 13, 2023
0fbdf44
ut pass
bigsheeper Dec 13, 2023
6fb9e86
ut pass2
bigsheeper Dec 14, 2023
cfb06ed
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 14, 2023
08bcf59
update
bigsheeper Dec 14, 2023
5ccad15
all test pass
bigsheeper Dec 14, 2023
1f0875e
all test pass 2
bigsheeper Dec 14, 2023
2d4f86f
update
bigsheeper Dec 14, 2023
7e28b25
all test pass 3
bigsheeper Dec 14, 2023
74300f1
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 14, 2023
4679afc
update
bigsheeper Dec 14, 2023
10f4199
update
bigsheeper Dec 14, 2023
b5a0588
feat: Add import reader for json
bigsheeper Dec 16, 2023
aaa146d
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 16, 2023
9815f1e
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 16, 2023
539c837
feat: Add import reader for numpy
bigsheeper Dec 16, 2023
6d7faf6
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 18, 2023
c97c26c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 18, 2023
925c38c
update
bigsheeper Dec 18, 2023
47e160c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 18, 2023
84eec47
code format
bigsheeper Dec 19, 2023
de37615
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 19, 2023
04c5fef
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 19, 2023
d799f47
code format
bigsheeper Dec 19, 2023
6fb736f
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 19, 2023
4061245
update
bigsheeper Dec 19, 2023
0fd267a
resolve conflicts
bigsheeper Dec 19, 2023
4d65b2d
refactor manager
bigsheeper Dec 19, 2023
cd8e524
update
bigsheeper Dec 19, 2023
fe853ca
update
bigsheeper Dec 20, 2023
f3066d2
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 20, 2023
4f75a24
add new func
bigsheeper Dec 20, 2023
4776c59
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 20, 2023
0687470
add log
bigsheeper Dec 20, 2023
240e82e
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 20, 2023
b6b2541
add import meta test
bigsheeper Dec 20, 2023
26122f0
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 20, 2023
8542be7
revert mock
bigsheeper Dec 20, 2023
a12b57d
use id allocator
bigsheeper Dec 21, 2023
9aef24b
add more ut
bigsheeper Dec 21, 2023
88fad66
code format
bigsheeper Dec 21, 2023
144fb24
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 21, 2023
20713ef
add import executor
bigsheeper Dec 23, 2023
c2a2c63
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 23, 2023
abb9b2f
update
bigsheeper Dec 23, 2023
878990b
add new func
bigsheeper Dec 23, 2023
c84c29e
add manager test
bigsheeper Dec 23, 2023
17a131c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 23, 2023
c037657
update hash func
bigsheeper Dec 23, 2023
0cc1017
update executor
bigsheeper Dec 23, 2023
f5e1df2
udpate
bigsheeper Dec 23, 2023
dbd46f8
add ut
bigsheeper Dec 23, 2023
f861eba
code format
bigsheeper Dec 23, 2023
1fa9c48
add mocker
bigsheeper Dec 23, 2023
a68ab86
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 23, 2023
64c9c7c
update proto
bigsheeper Dec 23, 2023
6a15df5
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 25, 2023
df1196e
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 25, 2023
b4908ad
update
bigsheeper Dec 25, 2023
c2fdeb1
update
bigsheeper Dec 25, 2023
b4f8237
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 25, 2023
bae2186
update
bigsheeper Dec 25, 2023
3ef72e1
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 25, 2023
f0bcd9c
update
bigsheeper Dec 25, 2023
f8b9ea3
resolve conflicts
bigsheeper Dec 25, 2023
32a5496
readPKs
bigsheeper Dec 25, 2023
ea516f0
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 25, 2023
ef0e987
update
bigsheeper Dec 25, 2023
043620d
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 25, 2023
d9d04a9
update
bigsheeper Dec 25, 2023
513e7ce
update
bigsheeper Dec 25, 2023
11062b8
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 25, 2023
819d26b
implement apis
bigsheeper Dec 25, 2023
ee1e2e4
implement import in proxt
bigsheeper Dec 26, 2023
63340a2
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 26, 2023
de63b81
code format
bigsheeper Dec 26, 2023
ecba12b
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 26, 2023
9f0d59c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 26, 2023
76c896c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 26, 2023
04d8882
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 26, 2023
579d935
update
bigsheeper Dec 26, 2023
6cb25b1
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 26, 2023
d7eb479
merge executor
bigsheeper Dec 26, 2023
5a3cedc
merge json reader
bigsheeper Dec 26, 2023
6189fed
Merge branch '2312-numpy-reader' of github.com:bigsheeper/milvus into…
bigsheeper Dec 26, 2023
072872c
Merge branch '2312-binlog-reader' of github.com:bigsheeper/milvus int…
bigsheeper Dec 26, 2023
e1e5355
Fix import
bigsheeper Dec 28, 2023
d71e839
add mock
bigsheeper Dec 28, 2023
50e1f80
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Dec 31, 2023
1526f36
ci tests pass 1
bigsheeper Jan 1, 2024
e76bca1
add parquet reader
bigsheeper Jan 1, 2024
fb49c87
code format
bigsheeper Jan 2, 2024
a9b55b8
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 2, 2024
88eda09
update proto
bigsheeper Jan 2, 2024
1394320
fix executor
bigsheeper Jan 2, 2024
79315c6
code format
bigsheeper Jan 2, 2024
2e57e0c
fix ut
bigsheeper Jan 2, 2024
2fa6883
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 3, 2024
bad9a1c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 3, 2024
f540137
add bulkinsert RESTful api
bigsheeper Jan 4, 2024
8fee759
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 4, 2024
a75299a
resolve conflicts
bigsheeper Jan 4, 2024
8817608
fix conflicts
bigsheeper Jan 4, 2024
5e2fb3c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 5, 2024
5da0093
merge binlog
bigsheeper Jan 5, 2024
d84f8b1
merge json
bigsheeper Jan 5, 2024
b5c3256
merge numpy
bigsheeper Jan 5, 2024
5dac277
merge parquet
bigsheeper Jan 5, 2024
f935f32
merge reader
bigsheeper Jan 5, 2024
aba4acd
merge restful
bigsheeper Jan 5, 2024
42852ce
fix reader in executor
bigsheeper Jan 5, 2024
a985d3f
fix error case
bigsheeper Jan 5, 2024
48b306c
merge binlog
bigsheeper Jan 5, 2024
0c5e061
remove count
bigsheeper Jan 5, 2024
b63ab03
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 5, 2024
f83d535
update hash
bigsheeper Jan 5, 2024
38e723e
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 5, 2024
9ba4b1e
merge parquet reader
bigsheeper Jan 8, 2024
64e2bb0
code format
bigsheeper Jan 8, 2024
1a2f44a
update ut
bigsheeper Jan 8, 2024
ee1aa49
GC import tasks
bigsheeper Jan 8, 2024
b3ffce7
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 8, 2024
f173fab
update logs
bigsheeper Jan 8, 2024
c0361c3
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 8, 2024
758c890
update pick segment policy
bigsheeper Jan 8, 2024
9bf629d
merge numpy reader
bigsheeper Jan 8, 2024
bcd14ea
resolve context and timeout
bigsheeper Jan 9, 2024
42e1f1a
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 9, 2024
37146a1
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 9, 2024
b8bd0ee
code format
bigsheeper Jan 9, 2024
47c1544
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 9, 2024
81d64fd
update checkErr
bigsheeper Jan 9, 2024
5684a89
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 9, 2024
f3d8e47
improve hash
bigsheeper Jan 9, 2024
a42758e
code format
bigsheeper Jan 9, 2024
11e1fb5
pass Options
bigsheeper Jan 9, 2024
78784b2
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 9, 2024
46cb4e1
update
bigsheeper Jan 9, 2024
6e19fec
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 9, 2024
54e4ebc
all test pass
bigsheeper Jan 10, 2024
8558b8e
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 10, 2024
4add758
fix ut
bigsheeper Jan 10, 2024
bbfe744
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 10, 2024
1e4d9e0
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 10, 2024
87258d7
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 10, 2024
0ed7411
debug parquet import
bigsheeper Jan 11, 2024
06d4d15
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 11, 2024
9e4cad3
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 11, 2024
313e00b
imporve parquet import performance
bigsheeper Jan 11, 2024
49cbe3d
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 11, 2024
4f18d44
add more integration test
bigsheeper Jan 12, 2024
f02b569
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 12, 2024
58a22df
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 12, 2024
8b45413
make params configurable
bigsheeper Jan 12, 2024
ea2c7b2
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 12, 2024
2a901cb
add monitoring
bigsheeper Jan 12, 2024
4175771
update estimate count policy
bigsheeper Jan 12, 2024
065042d
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 12, 2024
d28279f
fix RESTful api ut
bigsheeper Jan 12, 2024
b7e5087
update test dir
bigsheeper Jan 12, 2024
b1bbb36
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 12, 2024
8bf0bfa
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 16, 2024
b3893f6
fix ut
bigsheeper Jan 16, 2024
f999eeb
Fix large files import
bigsheeper Jan 18, 2024
acb32a5
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 18, 2024
785e63d
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 18, 2024
bb3b90d
fix rows update bug
bigsheeper Jan 19, 2024
330bf59
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 19, 2024
4341fe5
read concurrently
bigsheeper Jan 19, 2024
afefba3
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 19, 2024
c1d67a7
merge
bigsheeper Jan 26, 2024
44e9283
Fix mockery
bigsheeper Jan 26, 2024
1cf5fbd
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 26, 2024
df11548
update by code review
bigsheeper Jan 26, 2024
7105b10
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 26, 2024
11d5929
udpate
bigsheeper Jan 26, 2024
3ba5f6f
update by code review
bigsheeper Jan 29, 2024
8dbaf56
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 29, 2024
c079823
update
bigsheeper Jan 29, 2024
57b5203
code foramt
bigsheeper Jan 29, 2024
16d1423
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 29, 2024
de4a62e
update test
bigsheeper Jan 29, 2024
b6c72de
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 29, 2024
78f0d3d
list binlogs and group by segment
bigsheeper Jan 29, 2024
089551e
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 29, 2024
68da88e
Check if collection dropped
bigsheeper Jan 29, 2024
75c1265
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 29, 2024
ce170e8
add more ut and fix datanode import task concurrency
bigsheeper Jan 30, 2024
04386b4
fix conflicts
bigsheeper Jan 30, 2024
748dafd
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Jan 30, 2024
001169c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 2, 2024
0125098
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 4, 2024
7fa4d8b
enhance: Extend support for varchar autoID to BulkInsertV2
bigsheeper Feb 2, 2024
d52d985
update checkImportState
bigsheeper Feb 4, 2024
7c17159
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 4, 2024
01220f7
add more integration test
bigsheeper Feb 4, 2024
52621f4
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 4, 2024
89a4add
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 5, 2024
e3c3bb0
update by code review, 1. estimate by size, 2. seperate api in catalog
bigsheeper Feb 5, 2024
05f3342
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 18, 2024
7db69a5
add import jopb
bigsheeper Feb 18, 2024
7ebd1cf
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 18, 2024
5078623
gc import job
bigsheeper Feb 19, 2024
c35552b
simplify import task and preimport task
bigsheeper Feb 19, 2024
33ad8ed
add more ut
bigsheeper Feb 19, 2024
92d2bd8
add mockery
bigsheeper Feb 19, 2024
c7d7154
add mockery
bigsheeper Feb 19, 2024
92244be
fix ut and integration test
bigsheeper Feb 20, 2024
c04401c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 20, 2024
aa8b341
Merge branch '2312-import-debug' into 2312-import-scheduler
bigsheeper Feb 20, 2024
dc61824
fix httpserver ut
bigsheeper Feb 20, 2024
9446454
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 20, 2024
0eb4621
remove broker from import meta and add more ut
bigsheeper Feb 20, 2024
119ebb6
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 20, 2024
65f909a
revert unnecessary changes
bigsheeper Feb 20, 2024
6c80294
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 20, 2024
8bdaf9c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 20, 2024
4a32566
add more ut
bigsheeper Feb 21, 2024
88ec56b
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 21, 2024
8ef5cb6
revert test/python
bigsheeper Feb 21, 2024
801beb3
revert restful
bigsheeper Feb 21, 2024
06a688c
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 21, 2024
b93c076
remove api test
bigsheeper Feb 21, 2024
5206ef7
add more ut 2
bigsheeper Feb 21, 2024
70a68a8
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 21, 2024
859691b
Update by code review3
bigsheeper Feb 23, 2024
2961b07
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Feb 23, 2024
d3518d9
fix proxy client
bigsheeper Feb 23, 2024
75227ca
fix conflicts
bigsheeper Mar 1, 2024
447eada
fix by code review4
bigsheeper Mar 1, 2024
357b4e6
Merge branch 'master' of https://github.com/milvus-io/milvus into 231…
bigsheeper Mar 1, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' of https://github.com/milvus-io/milvus into 231… 
…2-import-scheduler
  • Loading branch information
@bigsheeper
bigsheeper committed Dec 23, 2023
commit a68ab86124abde7a83aec64239ef7503357f1ea9

This merge commit was added into this branch cleanly.

There are no new changes to show, but you can still view the diff.