[Serve] Can't autoscale deployment when target ongoing requests is 1

[spolcyn]

What happened + What you expected to happen

Issue: In Ray Serve, if target_num_ongoing_requests_per_replica is 1 and max_concurrent_queries is also 1, then autoscaling will never occur.

Expected: I can set the target to 1 and autoscaling will occur as I send more queries to the deployment, but a single replica will never have more than 1 ongoing request.

See: https://discuss.ray.io/t/autoscaling-with-max-concurrent-queries-1/6121 and the source of autoscaling_policy.py

Current workaround: Set max concurrent queries to 2, and target to 1. This increases request latency though, especially for queries that take a while (like model inference for large models).

Versions / Dependencies

Ray 1.12
Python/OS: N/A

Reproduction script

N/A

Issue Severity

Medium: It is a significant difficulty but I can work around it.

[spolcyn]

@simon-mo

[pandemosth]

This is an ongoing issue that impacts the usability of autoscaling.

My scenario is described in this issue for a related bug - #32652

I think the fundamental problem is that the autoscaling logic fails to consider current_handle_queued_queries and so only takes into account the number of in flight queries per replica:

ray/python/ray/serve/_private/autoscaling_policy.py

Lines 36 to 41 in 840215b

	# Example: if error_ratio == 2.0, we have two times too many ongoing
	# requests per replica, so we desire twice as many replicas.
	error_ratio: float = (
	num_ongoing_requests_per_replica
	/ autoscaling_config.target_num_ongoing_requests_per_replica
	)

As a result, if each replica continues to be busy with an in flight query no autoscaling will occur regardless of the number of queued queries.

[zcin]

This has been fixed now that we take queued requests into account for autoscaling.