how does `default_response_timeout` work?

[sreeprasannar]

📚 The doc issue

I set the value of default_response_timeout to 4 i.e. 4 seconds. At the start of the model load, this happens after 4 (ish) seconds:

org.pytorch.serve.wlm.WorkerInitializationException: Backend worker did not respond in given time

My guess is because the model takes a while to load (more than 4 seconds), the worker gets killed.
Is there a way to set a larger initial delay i.e. differentiate these two scenarios:

• account for the initial model load with a number different from default_response_timeout
• if model doesn't response in default_response_timeout after the initial load, then kill the worker

Suggest a potential alternative/fix

No response

[sreeprasannar]

I know I can set a higher default_response_timeout but that doesn't solve my problem. I want my model to go into UNLOADING state so that my kubernetes cluster can restart the pod. I want this to happen asap not after 120seconds hence the need to reduce the default_response_timeout

[sreeprasannar]

and a follow-up to this: is there a way to reduce the number of auto-recovery tries? Considering torchserve isn't really good at autorecovery anyway, I'd rather let kubernetes restart the pod

[arnavmehta7]

Agreed, yes, auto-recovery just stucks for me. Whereas restarting pod will be better.

[lxning]

@sreeprasannar @arnavmehta7 v0.8.0 allows for model level config (see model config yaml).

• increase responseTimeout in model config yaml file to handle long model loading latency.
• add maxRetryTimeoutInSec in model config yaml file to make ping return 500 and trigger pod restart. (see quota from v0.8.0 release notes => New Features => 10. Updated ping endpoint default behavior)

[arnavmehta7]

@lxning Does the responseTimeout consider the waiting time in queue? i.e., if my responseTimeout is 100 seconds, and a request is in queue for last 101 seconds, will it get cancelled?

[arnavmehta7]

Waiting time ns: 992559626073, Backend time ns: 14687643525
I am experiencing enormous Waiting times like 900seconds in this example. But quite less backend time :(