Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Serve] Revisiting ProxyState to fix draining sequence #41722

Merged
merged 29 commits into from
Jan 22, 2024
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
39c1447
Merge `start_new_ready_check` into `is_ready`
alexeykudinkin Dec 7, 2023
0231ccf
Cleaned up `try_update_status`;
alexeykudinkin Dec 7, 2023
0ebfe1b
Streamlined `is_healthy`, `is_ready` checks in `ProxyActorWrapper`
alexeykudinkin Dec 7, 2023
b6b3660
Tidying up
alexeykudinkin Dec 7, 2023
feffe80
Simplified `is_healthy`, `is_ready` checks even more
alexeykudinkin Dec 7, 2023
2144898
Fixed `is_drained` check to properly return whether proxy actor is pr…
alexeykudinkin Dec 7, 2023
d04d010
Streamlined proxy's state reconciliation flow
alexeykudinkin Dec 7, 2023
ab1cc40
Simplified readiness, health, drain checks;
alexeykudinkin Dec 8, 2023
5ba6068
Make sure exceptions are properly handled in proxy state reconciliation
alexeykudinkin Dec 8, 2023
d9ef34b
Cleaned up docs
alexeykudinkin Dec 8, 2023
e7c5e1e
Added test for ActorProxyWrapper
alexeykudinkin Dec 8, 2023
759d904
Fixed typos;
alexeykudinkin Dec 8, 2023
953343a
Fixed tests
alexeykudinkin Dec 8, 2023
73d881e
Fixed more tests
alexeykudinkin Dec 8, 2023
9e05832
Fixed some more tests
alexeykudinkin Dec 8, 2023
8d0cdcd
`lint`
alexeykudinkin Dec 8, 2023
136df1c
Fixing typo
alexeykudinkin Dec 8, 2023
189cad4
After rebase clean-up
alexeykudinkin Jan 10, 2024
c8dcb57
Tidying up
alexeykudinkin Jan 10, 2024
06b1755
Make proxy health-check configurable
alexeykudinkin Jan 11, 2024
f4283df
Unify `is_drained` API to require timeout be explicitly provided
alexeykudinkin Jan 11, 2024
b787188
Missing log statement
alexeykudinkin Jan 11, 2024
adfd2f3
Tidying up
alexeykudinkin Jan 18, 2024
eff6f23
Cleaned up exception handling
alexeykudinkin Jan 18, 2024
959feba
`_try_cancel_future` -> `_try_set_exception`
alexeykudinkin Jan 18, 2024
c5d2034
Make timeout_s optional
alexeykudinkin Jan 18, 2024
7002e11
Fixing typo
alexeykudinkin Jan 18, 2024
cc99b4a
Fixing another typo
alexeykudinkin Jan 18, 2024
8a1e954
Fixing tests
alexeykudinkin Jan 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Fixing tests
Signed-off-by: Alexey Kudinkin <ak@anyscale.com>
  • Loading branch information
alexeykudinkin committed Jan 20, 2024
commit 8a1e95420e9e3898e15358d531766cc93f66bffa
8 changes: 4 additions & 4 deletions python/ray/serve/tests/test_proxy_state.py
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,7 @@ def test_proxy_state_reconcile_draining_success():
timer.advance(5)

assert proxy_state._actor_proxy_wrapper.get_num_health_checks() == 10
assert proxy_state._actor_proxy_wrapper.get_num_drain_checks() == 9
assert proxy_state._actor_proxy_wrapper.get_num_drain_checks() == 10

# Make sure the status is still DRAINING
assert proxy_state.status == ProxyStatus.DRAINING
Expand All @@ -475,7 +475,7 @@ def test_proxy_state_reconcile_draining_success():

assert proxy_state._actor_proxy_wrapper.get_num_health_checks() == 15
# No new drain checks will occur, since there's a pending one (not completed yet)
assert proxy_state._actor_proxy_wrapper.get_num_drain_checks() == 14
assert proxy_state._actor_proxy_wrapper.get_num_drain_checks() == 15

# Simulate draining completed
proxy_state._actor_proxy_wrapper.is_drained_response = True
Expand Down Expand Up @@ -536,12 +536,12 @@ def test_proxy_actor_manager_removing_proxies(all_nodes, number_of_worker_nodes)
)
timer.advance(5)
# Assert that
# - Head-node proxy is HEALTHY
# - Head-node proxy is HEALTHY4
# - Worker node proxy is DRAINING
proxy_statuses = [manager._proxy_states[node_id].status for node_id in node_ids]
assert [ProxyStatus.HEALTHY, ProxyStatus.DRAINING] == proxy_statuses

assert worker_proxy_state._actor_proxy_wrapper.get_num_drain_checks() == N - 1
assert worker_proxy_state._actor_proxy_wrapper.get_num_drain_checks() == N

# Mark target proxy as fully drained
worker_proxy_state._actor_proxy_wrapper.is_drained_response = True
Expand Down
Loading