(raylet) [2023-05-08 23:36:25,536 C 69654 69654] plasma_store_provider.cc:75: Check failed: _s.ok() Bad status: IOError: Connection reset by peer

[scv119]

What happened + What you expected to happen

https://console.anyscale-staging.com/o/anyscale-internal/jobs/prodjob_27kmbw22tp9rnfp4g31q87mjdd

Versions / Dependencies

latest

Reproduction script

N/A

Issue Severity

High: It blocks me from completing my task.

[scv119]

(raylet) [2023-05-08 23:36:25,536 C 69654 69654] plasma_store_provider.cc:75:  Check failed: _s.ok() Bad status: IOError: Connection reset by peer
(raylet) *** StackTrace Information ***
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(+0xdb0cea) [0x7f72392a8cea] ray::operator<<()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(+0xdb27d2) [0x7f72392aa7d2] ray::SpdLogMessage::Flush()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(_ZN3ray6RayLogD1Ev+0x37) [0x7f72392aaae7] ray::RayLog::~RayLog()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(_ZN3ray4core29CoreWorkerPlasmaStoreProviderC2ERKSsSt10shared_ptrINS_6raylet12RayletClientEES4_INS0_16ReferenceCounterEESt8functionIFNS_6StatusEvEEbSA_IFSsvEE+0x4cc) [0x7f7238bfd7bc] ray::core::CoreWorkerPlasmaStoreProvider::CoreWorkerPlasmaStoreProvider()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(_ZN3ray4core10CoreWorkerC1ERKNS0_17CoreWorkerOptionsERKNS_8WorkerIDE+0x1acc) [0x7f7238ba09fc] ray::core::CoreWorker::CoreWorker()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(_ZN3ray4core21CoreWorkerProcessImplC2ERKNS0_17CoreWorkerOptionsE+0x587) [0x7f7238bacb57] ray::core::CoreWorkerProcessImpl::CoreWorkerProcessImpl()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(_ZN3ray4core17CoreWorkerProcess10InitializeERKNS0_17CoreWorkerOptionsE+0xd4) [0x7f7238badbb4] ray::core::CoreWorkerProcess::Initialize()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(+0x554994) [0x7f7238a4c994] __pyx_pf_3ray_7_raylet_10CoreWorker___cinit__()
(raylet) /home/ray/anaconda3/lib/python3.7/site-packages/ray/_raylet.so(+0x555a9d) [0x7f7238a4da9d] __pyx_tp_new_3ray_7_raylet_CoreWorker()
(raylet) ray::IDLE(_PyObject_FastCallKeywords+0xae) [0x4c25fe] _PyObject_FastCallKeywords
(raylet) ray::IDLE() [0x4ba1a9] call_function
(raylet) ray::IDLE(_PyEval_EvalFrameDefault+0x46c2) [0x4b6a12] _PyEval_EvalFrameDefault
(raylet) ray::IDLE(_PyEval_EvalCodeWithName+0x201) [0x4b10f1] _PyEval_EvalCodeWithName
(raylet) ray::IDLE(_PyFunction_FastCallKeywords+0x29c) [0x4c4f2c] _PyFunction_FastCallKeywords
(raylet) ray::IDLE() [0x4ba06f] call_function
(raylet) ray::IDLE(_PyEval_EvalFrameDefault+0x15e2) [0x4b3932] _PyEval_EvalFrameDefault
(raylet) ray::IDLE(_PyEval_EvalCodeWithName+0x201) [0x4b10f1] _PyEval_EvalCodeWithName
(raylet) ray::IDLE(PyEval_EvalCodeEx+0x39) [0x4b0ee9] PyEval_EvalCodeEx
(raylet) ray::IDLE(PyEval_EvalCode+0x1b) [0x54ee3b] PyEval_EvalCode
(raylet) ray::IDLE() [0x569ba3] run_mod
(raylet) ray::IDLE(PyRun_FileExFlags+0x97) [0x570377] PyRun_FileExFlags
(raylet) ray::IDLE(PyRun_SimpleFileExFlags+0x184) [0x56f934] PyRun_SimpleFileExFlags
(raylet) ray::IDLE() [0x544483] pymain_main
(raylet) ray::IDLE(_Py_UnixMain+0x3c) [0x54411c] _Py_UnixMain
(raylet) /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f7239e99083] __libc_start_main
(raylet) ray::IDLE() [0x543fce]
(raylet) 

[scv119]

Traceback (most recent call last):
  File "workloads/node_failures.py", line 60, in <module>
    ray.get(previous_ids)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/auto_init_hook.py", line 18, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/client_mode_hook.py", line 99, in wrapper
    return func(*args, **kwargs)
  File "/home/ray/anaconda3/lib/python3.7/site-packages/ray/_private/worker.py", line 2534, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError: ray::f() (pid=6120, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=42963, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=32204, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=3783, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=23083, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=32419, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=6177, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=32451, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=4218, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=4219, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=32353, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=6120, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=46671, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=4216, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=39115, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=68642, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=23464, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=46275, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=67091, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=46542, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=6120, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=38716, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=32520, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=68607, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=32314, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=58821, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=68644, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=23084, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=23371, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=3924, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=42411, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=23371, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=65287, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=68644, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=46281, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RayTaskError: ray::f() (pid=14162, ip=10.0.35.208)
  At least one of the input arguments for this task could not be computed:
ray.exceptions.RuntimeEnvSetupError: Failed to set up runtime environment.
Failed to request agent.
Subprocess return code: 1
[INFO 2023-05-08 23:36:35,381] anyscale_job_wra

[scv119]

Seems to me it's a bad timing where the raylet failure leads to worker's unexpected failure. My current take is this is unlikely a release blocker, but @rkooo567 can you help confirm?

[can-anyscale]

@scv119 : this test passed on the latest run https://buildkite.com/ray-project/release-tests-branch/builds/1650, can you help confirm and close the issue if thing has been resolved, thanks

[rkooo567]

yeah agreed with the assessment. It looks like if raylet fails during it is plasma store warmup, this can happen.

[rkooo567]

it's a little mysterious why it raises a runtime env setup failure, but I agree it is not a release blocker anyway...