Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add beta support to scale Stateful Executors with consensus using RAFT #5564

Merged
merged 165 commits into from
May 10, 2023
Merged

feat: add beta support to scale Stateful Executors with consensus using RAFT #5564

Changes from 1 commit
Commits
Show all changes
165 commits
Select commit Hold shift + click to select a range
6528743
feat: add snapshotting capacity
JoanFM Jan 2, 2023
98d036e
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 2, 2023
ea70e89
fix: remove merge problems
JoanFM Jan 2, 2023
4818b90
feat: rebuild protobuf
JoanFM Jan 2, 2023
fc60f69
style: fix overload and cli autocomplete
jina-bot Jan 2, 2023
bb6d9bc
fix: remove protobuf go
JoanFM Jan 2, 2023
d78e2bb
fix: remove changes in serve gateway
JoanFM Jan 2, 2023
11b412c
refactor: move endpoints to worker runtime
JoanFM Jan 3, 2023
2db7ce3
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 3, 2023
8a4810a
fix: fix test
JoanFM Jan 3, 2023
0bfb300
fix: snapshot in executor public
JoanFM Jan 3, 2023
43465e9
feat: add write decorator
JoanFM Jan 4, 2023
f2e811d
feat: add golang consensus
JoanFM Jan 5, 2023
23de4b0
feat: install with golang first good trial
JoanFM Jan 10, 2023
c8845f6
style: fix overload and cli autocomplete
jina-bot Jan 10, 2023
009af3c
feat: add more endpoints
JoanFM Jan 10, 2023
a9f247b
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 10, 2023
8cf5900
style: fix overload and cli autocomplete
jina-bot Jan 10, 2023
584d965
fix: restructure go package
JoanFM Jan 10, 2023
adb60a4
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Jan 10, 2023
a73775b
feat: manage the RAFT node
JoanFM Jan 11, 2023
ab30d2c
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 11, 2023
7c0153b
style: fix overload and cli autocomplete
jina-bot Jan 11, 2023
e7e7ca6
feat: properly add endpoint discovery and jina info rpcs
JoanFM Jan 11, 2023
8fb9d6a
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Jan 11, 2023
82756d4
feat: allow toy example to work
JoanFM Jan 12, 2023
7eb6176
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 12, 2023
f738c01
style: fix overload and cli autocomplete
jina-bot Jan 12, 2023
49af05b
feat: handle the add voter in orchestrator
JoanFM Jan 13, 2023
ac1d175
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Jan 13, 2023
b9b1141
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 13, 2023
98da89b
fix: try every replica when retrying
JoanFM Jan 16, 2023
529f2af
fix: make sure run raft only happens for workers
JoanFM Jan 16, 2023
3b0b47f
feat: handle terminate signal in golang
JoanFM Jan 16, 2023
18a1a06
feat: properly handle shards and workspaces
JoanFM Jan 16, 2023
f3560ee
feat: add raft configuration to arguments
JoanFM Jan 17, 2023
9b57c6a
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 17, 2023
94e39b2
style: fix overload and cli autocomplete
jina-bot Jan 17, 2023
e3e6aa1
fix: fix merge conflict
JoanFM Jan 17, 2023
2e2572b
fix: make golang extensions optional in setup
JoanFM Jan 17, 2023
129c316
style: fix overload and cli autocomplete
jina-bot Jan 17, 2023
1a2a715
fix: fix setup
JoanFM Jan 17, 2023
064f8cc
docs: pass commit lint
JoanFM Jan 17, 2023
bf2ab33
style: fix overload and cli autocomplete
jina-bot Jan 17, 2023
93e2697
fix: fix parsing raft config in golang
JoanFM Jan 17, 2023
529d560
fix: fix snapshoting directory
JoanFM Jan 18, 2023
09e5f9d
fix: fix snapshot
JoanFM Jan 18, 2023
1df3fb6
style: fix overload and cli autocomplete
jina-bot Jan 18, 2023
225a8ad
docs: fix docstrings
JoanFM Jan 18, 2023
1da4161
fix: remove unused file
JoanFM Jan 18, 2023
2501ffd
feat: add restore worker runtime executor part
JoanFM Jan 18, 2023
5bffd58
feat: implement snapshot and recover
JoanFM Jan 19, 2023
9f6c0fc
fix: fix the restore implementation
JoanFM Jan 19, 2023
8496a91
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 19, 2023
348b899
style: fix overload and cli autocomplete
jina-bot Jan 19, 2023
45f55e7
feat: add pod ports to guarantee restore
JoanFM Jan 19, 2023
d63899b
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Jan 19, 2023
5feecd4
style: fix overload and cli autocomplete
jina-bot Jan 19, 2023
5a583b4
feat: add toy script to play around with feature
JoanFM Jan 19, 2023
2601fbf
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Jan 19, 2023
44beb13
fix: fix demo
JoanFM Jan 19, 2023
9a971b9
Merge branch 'master' into jina-v-raft
JoanFM Jan 20, 2023
b565bec
refactor: small change toy stateful
JoanFM Jan 20, 2023
f0d864f
Merge branch 'master' into jina-v-raft
JoanFM Jan 24, 2023
ab8cafe
test: fix failing test
JoanFM Jan 24, 2023
3638de6
style: fix overload and cli autocomplete
jina-bot Jan 24, 2023
077ad02
Merge branch 'master' into jina-v-raft
JoanFM Jan 24, 2023
346b4a9
Merge branch 'master' into jina-v-raft
JoanFM Jan 24, 2023
557d7fb
Merge branch 'master' into jina-v-raft
JoanFM Jan 25, 2023
e388a98
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 25, 2023
6c501ba
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 26, 2023
ae6e0c9
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Jan 27, 2023
962e06e
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Feb 6, 2023
52b5575
fix: fix write decorator
JoanFM Feb 7, 2023
1ec12b7
Merge branch 'master' into jina-v-raft
JoanFM Feb 13, 2023
45fa322
style: fix overload and cli autocomplete
jina-bot Feb 13, 2023
c7a1dcb
Merge branch 'master' into jina-v-raft
JoanFM Feb 14, 2023
f1123bd
feat: implement health check rpc (#5721)
alaeddine-13 Mar 2, 2023
3f4676d
Merge branch 'master' into jina-v-raft
alaeddine-13 Mar 2, 2023
04acd8f
feat: easier setup (#5750)
alaeddine-13 Mar 21, 2023
298344d
fix: merge conflicts (#5773)
alaeddine-13 Mar 28, 2023
4e84914
Merge branch 'master' into jina-v-raft
alaeddine-13 Mar 29, 2023
b288984
fix: merge conflict
alaeddine-13 Mar 29, 2023
f79ce65
ci: build wheels for stateful executor jraft golang package (#5777)
alaeddine-13 Apr 3, 2023
8374815
Merge branch 'master' into jina-v-raft
JoanFM Apr 3, 2023
24b2416
fix: build and publish wheels in ci (#5783)
alaeddine-13 Apr 4, 2023
e6b221e
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Apr 11, 2023
f418f3f
style: fix overload and cli autocomplete
jina-bot Apr 11, 2023
e6cebd8
Merge branch 'master' into jina-v-raft
JoanFM Apr 12, 2023
21af643
Merge branch 'master' into jina-v-raft
JoanFM Apr 13, 2023
fef7991
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Apr 14, 2023
81b59db
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Apr 14, 2023
a9cf8b3
fix: adjust to new port
JoanFM Apr 14, 2023
ae843ef
test: add integration test for stateful executor (#5770)
alaeddine-13 Apr 17, 2023
d264b67
feat: add streaming RPC to golang
JoanFM Apr 17, 2023
20052f8
feat: clean logs
JoanFM Apr 18, 2023
c809554
fix: remove pod-ports argument
JoanFM Apr 18, 2023
649ad50
style: fix overload and cli autocomplete
jina-bot Apr 18, 2023
6a952ce
feat: first attempt to make it work with ContainerPod
JoanFM Apr 18, 2023
98ca5fb
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Apr 18, 2023
f3fa896
ci: add golang to Dockerfile
JoanFM Apr 18, 2023
dd7f552
test: add deployment as part of test
JoanFM Apr 18, 2023
fb8138e
Merge branch 'master' into jina-v-raft
JoanFM Apr 19, 2023
119b3a5
fix: fix head behavior with sharding
JoanFM Apr 19, 2023
3ca4b13
test: tests are runnable, jraft imported in different process
JoanFM Apr 19, 2023
8e4be36
style: fix overload and cli autocomplete
jina-bot Apr 19, 2023
03d4e6d
test: fix test
JoanFM Apr 19, 2023
45444e7
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Apr 19, 2023
92f0a20
test: remove useless test
JoanFM Apr 19, 2023
dda6c67
test: add tests for container stateful pod
JoanFM Apr 19, 2023
f87f17d
chore: add sanity check protocols and stateful flags
JoanFM Apr 19, 2023
cf97a3b
Merge branch 'master' into jina-v-raft
JoanFM Apr 19, 2023
7a07eee
style: fix overload and cli autocomplete
jina-bot Apr 19, 2023
0f84607
test: remove useless test
JoanFM Apr 19, 2023
66b0488
Merge branch 'master' into jina-v-raft
JoanFM Apr 20, 2023
f6e6d76
fix: fix restore with peer ports
JoanFM Apr 20, 2023
9aeeef7
test: do less tests
JoanFM Apr 20, 2023
17ed211
Merge branch 'master' into jina-v-raft
JoanFM Apr 21, 2023
dadd21a
feat: port stateful to docarray v2 (#5820)
JoanFM Apr 21, 2023
f830036
Merge branch 'master' into jina-v-raft
JoanFM Apr 24, 2023
44a6696
refactor: small changes, join raft worker
JoanFM Apr 24, 2023
84c8bcb
Merge branch 'master' into jina-v-raft
JoanFM Apr 24, 2023
46789e6
Merge branch 'master' of https://github.com/jina-ai/jina into jina-v-…
JoanFM Apr 24, 2023
70213d8
Merge branch 'master' into jina-v-raft
JoanFM Apr 25, 2023
f7b7a94
refactor: executor listen to the RAFT port + 100
JoanFM Apr 28, 2023
566bc7b
test: add test for sharding where merge is needed
JoanFM Apr 28, 2023
4bef03d
docs: add documentation about the stateful executor feature
JoanFM Apr 28, 2023
e045b28
docs: add documentation
JoanFM Apr 28, 2023
3110a1c
refactor: check some configurations
JoanFM Apr 28, 2023
8367849
style: fix overload and cli autocomplete
jina-bot Apr 28, 2023
0245a0a
feat: handle better parsing of `peer-ports`
JoanFM Apr 28, 2023
874cf09
Merge branch 'jina-v-raft' of https://github.com/jina-ai/jina into ji…
JoanFM Apr 28, 2023
9647efe
style: fix overload and cli autocomplete
jina-bot Apr 28, 2023
a77ff4a
Merge branch 'master' into jina-v-raft
JoanFM Apr 28, 2023
9f7381d
docs: add numbers of QPS in example
JoanFM May 2, 2023
19239de
docs: apply suggestions from code review
JoanFM May 2, 2023
664d556
Merge branch 'master' into jina-v-raft
JoanFM May 4, 2023
190cc16
Merge branch 'master' into jina-v-raft
JoanFM May 4, 2023
fe47a2e
ci: try to build wheels for 3.7
JoanFM May 5, 2023
035fd50
docs: apply suggestions from code review
JoanFM May 5, 2023
7804bb3
docs: update docs/concepts/orchestration/scale-out.md
JoanFM May 5, 2023
f765a79
Merge branch 'master' into jina-v-raft
JoanFM May 5, 2023
bbc4817
ci: integrate wheels building (#5845)
alaeddine-13 May 5, 2023
dcfbd32
ci: update code from jina-ai fork
JoanFM May 5, 2023
fec7517
chore: edit CI to consider wheels
JoanFM May 8, 2023
4a42989
Merge branch 'master' into jina-v-raft
JoanFM May 8, 2023
69c289e
ci: fix wheel finding
JoanFM May 8, 2023
f069c25
fix: fix merge conflicts
JoanFM May 8, 2023
9fbd025
fix: update request_handling.py
JoanFM May 8, 2023
fce7664
ci: change ci
JoanFM May 9, 2023
125a36d
fix: fix test with shards
JoanFM May 9, 2023
ecd31f8
fix: fix test stateful
JoanFM May 9, 2023
3291ed8
ci: stateful test independent
JoanFM May 9, 2023
886a37e
refactor: print when add voter fails
JoanFM May 9, 2023
1224c6d
style: fix overload and cli autocomplete
jina-bot May 9, 2023
7aac6cf
docs: apply suggestions from code review
JoanFM May 9, 2023
f9ea752
cd: adapt CD
JoanFM May 9, 2023
54f0018
test: mark test timeout
JoanFM May 10, 2023
d631101
test: skip test with restore for now
JoanFM May 10, 2023
1b11e74
chore: fix docstring
JoanFM May 10, 2023
a9ecadc
test: test with JINA_LOG_LEVEL
JoanFM May 10, 2023
d031d7b
style: fix overload and cli autocomplete
jina-bot May 10, 2023
ffe0f3e
cd: simplify cd yaml
JoanFM May 10, 2023
57a18f3
chore: add more debug info
JoanFM May 10, 2023
e10211c
style: fix overload and cli autocomplete
jina-bot May 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into jina-v-raft
  • Loading branch information
@JoanFM
JoanFM authored Apr 21, 2023
commit 17ed21199d1b3d4ba6e1d5e0c7fc591d003c9f3b

This merge commit was added into this branch cleanly.

There are no new changes to show, but you can still view the diff.