pandas read_csv : large file failed? #2209
Description
System information
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
Ubuntu 18.04 - Modin version (
modin.__version__):
0.8.1.1 - Python version:
3.8.5 - Code we can use to reproduce:
May be read a large file (~200G), and it doesn't matter withdtype
Describe the problem
pandas read_csv failed
Source code / logs
d=14047) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14047) PC: @ 0x0 (unknown)
(pid=14047) *** SIGBUS (@0x7e9ee0beefe0) received by PID 14047 (TID 0x7f2528e9a740) from PID 18446744073185193952; stack trace: ***
(pid=14047) @ 0x7f2528ab28a0 (unknown)
(pid=14047) @ 0x7f252883e0d3 (unknown)
(pid=14047) @ 0x7f25268442d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=14047) @ 0x7f252683c90b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=14047) @ 0x7f2526875b8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=14047) @ 0x7f25268c0d5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=14047) @ 0x7f25268937d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=14047) @ 0x7f252686d72c std::_Function_handler<>::_M_invoke()
(pid=14047) @ 0x7f2526944882 ray::CoreWorker::ExecuteTask()
(pid=14047) @ 0x7f25268d76cb std::_Function_handler<>::_M_invoke()
(pid=13971) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13971) PC: @ 0x0 (unknown)
(pid=13971) *** SIGBUS (@0x7ea7147b6000) received by PID 13971 (TID 0x7e99f49fb700) from PID 343629824; stack trace: ***
(pid=13971) @ 0x7f31361ed8a0 (unknown)
(pid=13971) @ 0x7f3135f78d8f (unknown)
(pid=13971) @ 0x7f3133b5e421 execute_native_thread_routine_compat
(pid=13971) @ 0x7f31361e26db start_thread
(pid=13971) @ 0x7f3135f0ba3f clone
(pid=14071) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14071) PC: @ 0x0 (unknown)
(pid=14071) *** SIGBUS (@0x7f6eaf436fe0) received by PID 14071 (TID 0x7ff42ed48740) from PID 18446744072355016672; stack trace: ***
(pid=14071) @ 0x7ff42e9608a0 (unknown)
(pid=14071) @ 0x7ff42e6ec0d3 (unknown)
(pid=14071) @ 0x7ff42c6f22d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=14071) @ 0x7ff42c6ea90b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=14071) @ 0x7ff42c723b8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=14071) @ 0x7ff42c76ed5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=14071) @ 0x7ff42c7417d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=14071) @ 0x7ff42c71b72c std::_Function_handler<>::_M_invoke()
(pid=14071) @ 0x7ff42c7f2882 ray::CoreWorker::ExecuteTask()
(pid=14071) @ 0x7ff42c7856cb std::_Function_handler<>::_M_invoke()
(pid=14071) @ 0x7ff42c80fced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=13986) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13986) PC: @ 0x0 (unknown)
(pid=13986) *** SIGBUS (@0x7f566ecfe000) received by PID 13986 (TID 0x7f48411fc700) from PID 1859117056; stack trace: ***
(pid=13986) @ 0x7fdf826038a0 (unknown)
(pid=13986) @ 0x7fdf8238ed8f (unknown)
(pid=13986) @ 0x7fdf7ff74421 execute_native_thread_routine_compat
(pid=13986) @ 0x7fdf825f86db start_thread
(pid=13986) @ 0x7fdf82321a3f clone
(pid=13967) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13967) PC: @ 0x0 (unknown)
(pid=13967) *** SIGBUS (@0x7f3a5647d000) received by PID 13967 (TID 0x7f2f241fa700) from PID 1447546880; stack trace: ***
(pid=13967) @ 0x7fc6685fe8a0 (unknown)
(pid=13967) @ 0x7fc668389d8f (unknown)
(pid=13967) @ 0x7fc665f6f421 execute_native_thread_routine_compat
(pid=13967) @ 0x7fc6685f36db start_thread
(pid=13967) @ 0x7fc66831ca3f clone
(pid=14039) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14039) PC: @ 0x0 (unknown)
(pid=14039) *** SIGBUS (@0x7f31fa991000) received by PID 14039 (TID 0x7f223effd700) from PID 18446744073618919424; stack trace: ***
(pid=14039) @ 0x7fb9811258a0 (unknown)
(pid=14039) @ 0x7fb980eb0d8f (unknown)
(pid=14039) @ 0x7fb97ea96421 execute_native_thread_routine_compat
(pid=14039) @ 0x7fb98111a6db start_thread
(pid=14039) @ 0x7fb980e43a3f clone
(pid=14056) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14056) PC: @ 0x0 (unknown)
(pid=14056) *** SIGBUS (@0x7e964d933000) received by PID 14056 (TID 0x7e88d7932700) from PID 1301491712; stack trace: ***
(pid=14056) @ 0x7f201ccd68a0 (unknown)
(pid=14056) @ 0x7f201ca61d8f (unknown)
(pid=14056) @ 0x7f201a647421 execute_native_thread_routine_compat
(pid=14056) @ 0x7f201cccb6db start_thread
(pid=14056) @ 0x7f201c9f4a3f clone
(pid=13959) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13959) PC: @ 0x0 (unknown)
(pid=13959) *** SIGBUS (@0x7f3fb9cab000) received by PID 13959 (TID 0x7f31d41fa700) from PID 18446744072531652608; stack trace: ***
(pid=13959) @ 0x7fc916c1d8a0 (unknown)
(pid=13959) @ 0x7fc9169a8d8f (unknown)
(pid=13959) @ 0x7fc91458e421 execute_native_thread_routine_compat
(pid=13959) @ 0x7fc916c126db start_thread
(pid=13959) @ 0x7fc91693ba3f clone
(pid=13962) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13962) PC: @ 0x0 (unknown)
(pid=13962) *** SIGBUS (@0x7e7d27019000) received by PID 13962 (TID 0x7e6ebe9f7700) from PID 654413824; stack trace: ***
(pid=13962) @ 0x7f06038238a0 (unknown)
(pid=13962) @ 0x7f06035aed8f (unknown)
(pid=13962) @ 0x7f0601194421 execute_native_thread_routine_compat
(pid=13962) @ 0x7f06038186db start_thread
(pid=13962) @ 0x7f0603541a3f clone
(pid=13970) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13970) PC: @ 0x0 (unknown)
(pid=13970) *** SIGBUS (@0x7f55e65b8fe0) received by PID 13970 (TID 0x7fdadde74740) from PID 18446744073279344608; stack trace: ***
(pid=13970) @ 0x7fdadda8c8a0 (unknown)
(pid=13970) @ 0x7fdadd8180df (unknown)
(pid=13970) @ 0x7fdadb81e2d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=13970) @ 0x7fdadb81690b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=13970) @ 0x7fdadb84fb8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=13970) @ 0x7fdadb89ad5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=13970) @ 0x7fdadb86d7d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=13970) @ 0x7fdadb84772c std::_Function_handler<>::_M_invoke()
(pid=13970) @ 0x7fdadb91e882 ray::CoreWorker::ExecuteTask()
(pid=13970) @ 0x7fdadb8b16cb std::_Function_handler<>::_M_invoke()
(pid=13970) @ 0x7fdadb93bced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=14100) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14100) PC: @ 0x0 (unknown)
(pid=14100) *** SIGBUS (@0x7e8476212000) received by PID 14100 (TID 0x7e74ff9f9700) from PID 1981882368; stack trace: ***
(pid=14100) @ 0x7f0c423b38a0 (unknown)
(pid=14100) @ 0x7f0c4213ed8f (unknown)
(pid=14100) @ 0x7f0c3fd24421 execute_native_thread_routine_compat
(pid=14100) @ 0x7f0c423a86db start_thread
(pid=14100) @ 0x7f0c420d1a3f clone
(pid=13990) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13990) PC: @ 0x0 (unknown)
(pid=13990) *** SIGBUS (@0x7e9dc01f4fe0) received by PID 13990 (TID 0x7f238366c740) from PID 18446744072637861856; stack trace: ***
(pid=13990) @ 0x7f23832848a0 (unknown)
(pid=13990) @ 0x7f23830100d9 (unknown)
(pid=13990) @ 0x7f23810162d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=13990) @ 0x7f238100e90b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=13990) @ 0x7f2381047b8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=13990) @ 0x7f2381092d5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=13990) @ 0x7f23810657d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=13990) @ 0x7f238103f72c std::_Function_handler<>::_M_invoke()
(pid=13990) @ 0x7f2381116882 ray::CoreWorker::ExecuteTask()
(pid=13990) @ 0x7f23810a96cb std::_Function_handler<>::_M_invoke()
(pid=13990) @ 0x7f2381133ced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=13990) @ 0x7f2381131fe3 ray::SchedulingQueue::Add()
(pid=13990) @ 0x7f2381135030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=14092) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14092) PC: @ 0x0 (unknown)
(pid=14092) *** SIGBUS (@0x7f4d40a3f000) received by PID 14092 (TID 0x7f40da9f7700) from PID 1084485632; stack trace: ***
(pid=14092) @ 0x7fd81f2838a0 (unknown)
(pid=14092) @ 0x7fd81f00ed8f (unknown)
(pid=14092) @ 0x7fd81cbf4421 execute_native_thread_routine_compat
(pid=14092) @ 0x7fd81f2786db start_thread
(pid=14092) @ 0x7fd81efa1a3f clone
(pid=14037) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14037) PC: @ 0x0 (unknown)
(pid=14037) *** SIGBUS (@0x7ed9796a1fe0) received by PID 14037 (TID 0x7f5e351c1740) from PID 2036998112; stack trace: ***
(pid=14037) @ 0x7f5e34dd98a0 (unknown)
(pid=14037) @ 0x7f5e34b650d3 (unknown)
(pid=14037) @ 0x7f5e32b6b2d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=14037) @ 0x7f5e32b6390b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=14037) @ 0x7f5e32b9cb8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=14037) @ 0x7f5e32be7d5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=14037) @ 0x7f5e32bba7d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=14037) @ 0x7f5e32b9472c std::_Function_handler<>::_M_invoke()
(pid=14037) @ 0x7f5e32c6b882 ray::CoreWorker::ExecuteTask()
(pid=14037) @ 0x7f5e32bfe6cb std::_Function_handler<>::_M_invoke()
(pid=14037) @ 0x7f5e32c88ced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=14037) @ 0x7f5e32c86fe3 ray::SchedulingQueue::Add()
(pid=14037) @ 0x7f5e32c8a030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=14033) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14033) PC: @ 0x0 (unknown)
(pid=14033) *** SIGBUS (@0x7e8c0045bfe0) received by PID 14033 (TID 0x7f102e60f740) from PID 4571104; stack trace: ***
(pid=14033) @ 0x7f102e2278a0 (unknown)
(pid=14033) @ 0x7f102dfb30d9 (unknown)
(pid=14033) @ 0x7f102bfb92d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=14033) @ 0x7f102bfb190b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=14033) @ 0x7f102bfeab8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=14033) @ 0x7f102c035d5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=14033) @ 0x7f102c0087d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=14033) @ 0x7f102bfe272c std::_Function_handler<>::_M_invoke()
(pid=14033) @ 0x7f102c0b9882 ray::CoreWorker::ExecuteTask()
(pid=14033) @ 0x7f102c04c6cb std::_Function_handler<>::_M_invoke()
(pid=14033) @ 0x7f102c0d6ced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=14033) @ 0x7f102c0d4fe3 ray::SchedulingQueue::Add()
(pid=14033) @ 0x7f102c0d8030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=13977) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13977) PC: @ 0x0 (unknown)
(pid=13977) *** SIGBUS (@0x7f7020a68fe0) received by PID 13977 (TID 0x7ff625f72740) from PID 547786720; stack trace: ***
(pid=13977) @ 0x7ff625b8a8a0 (unknown)
(pid=13977) @ 0x7ff6259160df (unknown)
(pid=13977) @ 0x7ff62391c2d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=13977) @ 0x7ff62391490b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=13977) @ 0x7ff62394db8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=13977) @ 0x7ff623998d5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=13977) @ 0x7ff62396b7d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=13977) @ 0x7ff62394572c std::_Function_handler<>::_M_invoke()
(pid=13977) @ 0x7ff623a1c882 ray::CoreWorker::ExecuteTask()
(pid=13977) @ 0x7ff6239af6cb std::_Function_handler<>::_M_invoke()
(pid=13977) @ 0x7ff623a39ced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=13977) @ 0x7ff623a37fe3 ray::SchedulingQueue::Add()
(pid=13977) @ 0x7ff623a3b030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=14076) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14076) PC: @ 0x0 (unknown)
(pid=14076) *** SIGBUS (@0x7ee08f134fe0) received by PID 14076 (TID 0x7f65c7f33740) from PID 18446744071814991840; stack trace: ***
(pid=14076) @ 0x7f65c7b4b8a0 (unknown)
(pid=14076) @ 0x7f65c78d70d9 (unknown)
(pid=14076) @ 0x7f65c58dd2d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=14076) @ 0x7f65c58d590b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=14076) @ 0x7f65c590eb8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=14076) @ 0x7f65c5959d5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=14076) @ 0x7f65c592c7d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=14076) @ 0x7f65c590672c std::_Function_handler<>::_M_invoke()
(pid=14076) @ 0x7f65c59dd882 ray::CoreWorker::ExecuteTask()
(pid=14076) @ 0x7f65c59706cb std::_Function_handler<>::_M_invoke()
(pid=14076) @ 0x7f65c59faced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=14076) @ 0x7f65c59f8fe3 ray::SchedulingQueue::Add()
(pid=14076) @ 0x7f65c59fc030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=14085) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14085) PC: @ 0x0 (unknown)
(pid=14085) *** SIGBUS (@0x7f4ba6e86fe0) received by PID 14085 (TID 0x7fcf4d2ca740) from PID 18446744072214835168; stack trace: ***
(pid=14085) @ 0x7fcf4cee28a0 (unknown)
(pid=14085) @ 0x7fcf4cc6e0d9 (unknown)
(pid=14085) @ 0x7fcf4ac742d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=14085) @ 0x7fcf4ac6c90b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=14085) @ 0x7fcf4aca5b8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=14085) @ 0x7fcf4acf0d5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=14085) @ 0x7fcf4acc37d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=14085) @ 0x7fcf4ac9d72c std::_Function_handler<>::_M_invoke()
(pid=14085) @ 0x7fcf4ad74882 ray::CoreWorker::ExecuteTask()
(pid=14085) @ 0x7fcf4ad076cb std::_Function_handler<>::_M_invoke()
(pid=14085) @ 0x7fcf4ad91ced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=14085) @ 0x7fcf4ad8ffe3 ray::SchedulingQueue::Add()
(pid=14085) @ 0x7fcf4ad93030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=13965) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13965) PC: @ 0x0 (unknown)
(pid=13965) *** SIGBUS (@0x7ea3a4db0000) received by PID 13965 (TID 0x7e98037fe700) from PID 18446744072180400128; stack trace: ***
(pid=13965) @ 0x7f2f452868a0 (unknown)
(pid=13965) @ 0x7f2f45011d8f (unknown)
(pid=13965) @ 0x7f2f42bf7421 execute_native_thread_routine_compat
(pid=13965) @ 0x7f2f4527b6db start_thread
(pid=13965) @ 0x7f2f44fa4a3f clone
(pid=14088) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14088) PC: @ 0x0 (unknown)
(pid=14088) *** SIGBUS (@0x7eb15a5ab000) received by PID 14088 (TID 0x7ea2ac9fb700) from PID 1515892736; stack trace: ***
(pid=14088) @ 0x7f39eea0b8a0 (unknown)
(pid=14088) @ 0x7f39ee796d8f (unknown)
(pid=14088) @ 0x7f39ec37c421 execute_native_thread_routine_compat
(pid=14088) @ 0x7f39eea006db start_thread
(pid=14088) @ 0x7f39ee729a3f clone
(pid=13979) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13979) PC: @ 0x0 (unknown)
(pid=13979) *** SIGBUS (@0x7eec535df000) received by PID 13979 (TID 0x7edbe09fb700) from PID 1398665216; stack trace: ***
(pid=13979) @ 0x7f731f34b8a0 (unknown)
(pid=13979) @ 0x7f731f0d6d8f (unknown)
(pid=13979) @ 0x7f731ccbc421 execute_native_thread_routine_compat
(pid=13979) @ 0x7f731f3406db start_thread
(pid=13979) @ 0x7f731f069a3f clone
(pid=14031) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14031) PC: @ 0x0 (unknown)
(pid=14031) *** SIGBUS (@0x7ecb51c94000) received by PID 14031 (TID 0x7ebff47ca700) from PID 1372143616; stack trace: ***
(pid=14031) @ 0x7f5735ca48a0 (unknown)
(pid=14031) @ 0x7f5735a2fd8f (unknown)
(pid=14031) @ 0x7f5733615421 execute_native_thread_routine_compat
(pid=14031) @ 0x7f5735c996db start_thread
(pid=14031) @ 0x7f57359c2a3f clone
(pid=13969) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13969) PC: @ 0x0 (unknown)
(pid=13969) *** SIGBUS (@0x7f6300c30fe0) received by PID 13969 (TID 0x7fe770596740) from PID 12783584; stack trace: ***
(pid=13969) @ 0x7fe7701ae8a0 (unknown)
(pid=13969) @ 0x7fe76ff3a0d3 (unknown)
(pid=13969) @ 0x7fe76df402d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=13969) @ 0x7fe76df3890b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=13969) @ 0x7fe76df71b8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=13969) @ 0x7fe76dfbcd5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=13969) @ 0x7fe76df8f7d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=13969) @ 0x7fe76df6972c std::_Function_handler<>::_M_invoke()
(pid=13969) @ 0x7fe76e040882 ray::CoreWorker::ExecuteTask()
(pid=13969) @ 0x7fe76dfd36cb std::_Function_handler<>::_M_invoke()
(pid=13969) @ 0x7fe76e05dced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=13969) @ 0x7fe76e05bfe3 ray::SchedulingQueue::Add()
(pid=13969) @ 0x7fe76e05f030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=13969) @ 0x7fe76e02208b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14035) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14035) PC: @ 0x0 (unknown)
(pid=14035) *** SIGBUS (@0x7f6b77506000) received by PID 14035 (TID 0x7f5f89ffb700) from PID 2001756160; stack trace: ***
(pid=14035) @ 0x7ff6ccfed8a0 (unknown)
(pid=14035) @ 0x7ff6ccd78d8f (unknown)
(pid=14035) @ 0x7ff6ca95e421 execute_native_thread_routine_compat
(pid=14035) @ 0x7ff6ccfe26db start_thread
(pid=14035) @ 0x7ff6ccd0ba3f clone
(pid=14160) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14160) PC: @ 0x0 (unknown)
(pid=14160) *** SIGBUS (@0x7ed20cc29000) received by PID 14160 (TID 0x7ec5e67fc700) from PID 214077440; stack trace: ***
(pid=14160) @ 0x7f5d296628a0 (unknown)
(pid=14160) @ 0x7f5d293edd8f (unknown)
(pid=14160) @ 0x7f5d26fd3421 execute_native_thread_routine_compat
(pid=14160) @ 0x7f5d296576db start_thread
(pid=14160) @ 0x7f5d29380a3f clone
(pid=14049) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14049) PC: @ 0x0 (unknown)
(pid=14049) *** SIGBUS (@0x7ec40b6e2fe0) received by PID 14049 (TID 0x7f47f5ac5740) from PID 191770592; stack trace: ***
(pid=14049) @ 0x7f47f56dd8a0 (unknown)
(pid=14049) @ 0x7f47f54690df (unknown)
(pid=14049) @ 0x7f47f346f2d6 __pyx_f_3ray_7_raylet_13Pickle5Writer_write_to()
(pid=14049) @ 0x7f47f346790b __pyx_f_3ray_7_raylet_23Pickle5SerializedObject_write_to()
(pid=14049) @ 0x7f47f34a0b8b __pyx_f_3ray_7_raylet_27MessagePackSerializedObject_write_to()
(pid=14049) @ 0x7f47f34ebd5f __pyx_f_3ray_7_raylet_10CoreWorker_store_task_outputs()
(pid=14049) @ 0x7f47f34be7d8 __pyx_f_3ray_7_raylet_task_execution_handler()
(pid=14049) @ 0x7f47f349872c std::_Function_handler<>::_M_invoke()
(pid=14049) @ 0x7f47f356f882 ray::CoreWorker::ExecuteTask()
(pid=14049) @ 0x7f47f35026cb std::_Function_handler<>::_M_invoke()
(pid=14049) @ 0x7f47f358cced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=14049) @ 0x7f47f358afe3 ray::SchedulingQueue::Add()
(pid=14049) @ 0x7f47f358e030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=14049) @ 0x7f47f355108b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14105) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14105) PC: @ 0x0 (unknown)
(pid=14105) *** SIGBUS (@0x7f674502c000) received by PID 14105 (TID 0x7f5709ffb700) from PID 1157808128; stack trace: ***
(pid=14105) @ 0x7fee4dca38a0 (unknown)
(pid=14105) @ 0x7fee4da2ed8f (unknown)
(pid=14105) @ 0x7fee4b614421 execute_native_thread_routine_compat
(pid=14105) @ 0x7fee4dc986db start_thread
(pid=14105) @ 0x7fee4d9c1a3f clone
(pid=14113) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14113) PC: @ 0x0 (unknown)
(pid=14113) *** SIGBUS (@0x7f2f87b8f000) received by PID 14113 (TID 0x7f20951fc700) from PID 18446744071691628544; stack trace: ***
(pid=14113) @ 0x7fb7d7de98a0 (unknown)
(pid=14113) @ 0x7fb7d7b74d8f (unknown)
(pid=14113) @ 0x7fb7d575a421 execute_native_thread_routine_compat
(pid=14113) @ 0x7fb7d7dde6db start_thread
(pid=14113) @ 0x7fb7d7b07a3f clone
(pid=14026) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14026) PC: @ 0x0 (unknown)
(pid=14026) *** SIGBUS (@0x7ee7cbb90000) received by PID 14026 (TID 0x7edcf01fa700) from PID 18446744072832483328; stack trace: ***
(pid=14026) @ 0x7f74322798a0 (unknown)
(pid=14026) @ 0x7f7432004d8f (unknown)
(pid=14026) @ 0x7f742fbea421 execute_native_thread_routine_compat
(pid=14026) @ 0x7f743226e6db start_thread
(pid=14026) @ 0x7f7431f97a3f clone
(pid=14159) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14159) PC: @ 0x0 (unknown)
(pid=14159) *** SIGBUS (@0x7f434b80e000) received by PID 14159 (TID 0x7f32901fa700) from PID 1266737152; stack trace: ***
(pid=14159) @ 0x7fc9d396b8a0 (unknown)
(pid=14159) @ 0x7fc9d36f6d8f (unknown)
(pid=14159) @ 0x7fc9d12dc421 execute_native_thread_routine_compat
(pid=14159) @ 0x7fc9d39606db start_thread
(pid=14159) @ 0x7fc9d3689a3f clone
(pid=14074) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14074) PC: @ 0x0 (unknown)
(pid=14074) *** SIGBUS (@0x7edf2d611000) received by PID 14074 (TID 0x7ed24a9f7700) from PID 761335808; stack trace: ***
(pid=14074) @ 0x7f698fa868a0 (unknown)
(pid=14074) @ 0x7f698f811d8f (unknown)
(pid=14074) @ 0x7f698d3f7421 execute_native_thread_routine_compat
(pid=14074) @ 0x7f698fa7b6db start_thread
(pid=14074) @ 0x7f698f7a4a3f clone
(pid=14098) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14098) PC: @ 0x0 (unknown)
(pid=14098) *** SIGBUS (@0x7e74f0b76000) received by PID 14098 (TID 0x7e64ec9fb700) from PID 18446744073453133824; stack trace: ***
(pid=14098) @ 0x7efc2fa0f8a0 (unknown)
(pid=14098) @ 0x7efc2f79ad8f (unknown)
(pid=14098) @ 0x7efc2d380421 execute_native_thread_routine_compat
(pid=14098) @ 0x7efc2fa046db start_thread
(pid=14098) @ 0x7efc2f72da3f clone
(pid=14050) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14050) PC: @ 0x0 (unknown)
(pid=14050) *** SIGBUS (@0x7ec2b5dcd000) received by PID 14050 (TID 0x7eb5071f8700) from PID 18446744072465731584; stack trace: ***
(pid=14050) @ 0x7f4c4ad178a0 (unknown)
(pid=14050) @ 0x7f4c4aaa2d8f (unknown)
(pid=14050) @ 0x7f4c48688421 execute_native_thread_routine_compat
(pid=14050) @ 0x7f4c4ad0c6db start_thread
(pid=14050) @ 0x7f4c4aa35a3f clone
(pid=14084) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14084) PC: @ 0x0 (unknown)
(pid=14084) *** SIGBUS (@0x7ef7bafd3000) received by PID 14084 (TID 0x7ee8867fc700) from PID 18446744072551739392; stack trace: ***
(pid=14084) @ 0x7f7fc8b058a0 (unknown)
(pid=14084) @ 0x7f7fc8890d8f (unknown)
(pid=14084) @ 0x7f7fc6476421 execute_native_thread_routine_compat
(pid=14084) @ 0x7f7fc8afa6db start_thread
(pid=14084) @ 0x7f7fc8823a3f clone
(pid=13982) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=13982) PC: @ 0x0 (unknown)
(pid=13982) *** SIGBUS (@0x7e72f69bf000) received by PID 13982 (TID 0x7e68511fc700) from PID 18446744073551998976; stack trace: ***
(pid=13982) @ 0x7eff930148a0 (unknown)
(pid=13982) @ 0x7eff92d9fd8f (unknown)
(pid=13982) @ 0x7eff90985421 execute_native_thread_routine_compat
(pid=13982) @ 0x7eff930096db start_thread
(pid=13982) @ 0x7eff92d32a3f clone
(pid=14020) *** Aborted at 1601696529 (unix time) try "date -d @1601696529" if you are using GNU date ***
(pid=14020) PC: @ 0x0 (unknown)
(pid=14020) *** SIGBUS (@0x7f0ce8d81000) received by PID 14020 (TID 0x7f003c9fb700) from PID 18446744073321058304; stack trace: ***
(pid=14020) @ 0x7f977fb358a0 (unknown)
(pid=14020) @ 0x7f977f8c0d8f (unknown)
(pid=14020) @ 0x7f977d4a6421 execute_native_thread_routine_compat
(pid=14020) @ 0x7f977fb2a6db start_thread
(pid=14020) @ 0x7f977f853a3f clone
(pid=14047) @ 0x7f2526961ced _ZZN3ray28CoreWorkerDirectTaskReceiver14HandlePushTaskERKNS_3rpc15PushTaskRequestEPNS1_13PushTaskReplyESt8functionIFvNS_6StatusES7_IFvvEESA_EEENKUlvE_clEv
(pid=14047) @ 0x7f252695ffe3 ray::SchedulingQueue::Add()
(pid=14047) @ 0x7f2526963030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=14047) @ 0x7f252692608b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14047) @ 0x7f2526d1fc3f boost::asio::detail::scheduler::do_run_one()
(pid=14047) @ 0x7f2526d20a01 boost::asio::detail::scheduler::run()
(pid=14047) @ 0x7f2526d21952 boost::asio::io_context::run()
(pid=14071) @ 0x7ff42c80dfe3 ray::SchedulingQueue::Add()
(pid=14071) @ 0x7ff42c811030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=14071) @ 0x7ff42c7d408b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14071) @ 0x7ff42cbcdc3f boost::asio::detail::scheduler::do_run_one()
(pid=14071) @ 0x7ff42cbcea01 boost::asio::detail::scheduler::run()
(pid=14071) @ 0x7ff42cbcf952 boost::asio::io_context::run()
(pid=13970) @ 0x7fdadb939fe3 ray::SchedulingQueue::Add()
(pid=13970) @ 0x7fdadb93d030 ray::CoreWorkerDirectTaskReceiver::HandlePushTask()
(pid=13970) @ 0x7fdadb90008b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=13970) @ 0x7fdadbcf9c3f boost::asio::detail::scheduler::do_run_one()
(pid=13970) @ 0x7fdadbcfaa01 boost::asio::detail::scheduler::run()
(pid=13970) @ 0x7fdadbcfb952 boost::asio::io_context::run()
(pid=13990) @ 0x7f23810f808b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=13990) @ 0x7f23814f1c3f boost::asio::detail::scheduler::do_run_one()
(pid=13990) @ 0x7f23814f2a01 boost::asio::detail::scheduler::run()
(pid=13990) @ 0x7f23814f3952 boost::asio::io_context::run()
(pid=14037) @ 0x7f5e32c4d08b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14037) @ 0x7f5e33046c3f boost::asio::detail::scheduler::do_run_one()
(pid=14037) @ 0x7f5e33047a01 boost::asio::detail::scheduler::run()
(pid=14037) @ 0x7f5e33048952 boost::asio::io_context::run()
(pid=14033) @ 0x7f102c09b08b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14033) @ 0x7f102c494c3f boost::asio::detail::scheduler::do_run_one()
(pid=14033) @ 0x7f102c495a01 boost::asio::detail::scheduler::run()
(pid=14033) @ 0x7f102c496952 boost::asio::io_context::run()
(pid=13977) @ 0x7ff6239fe08b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=13977) @ 0x7ff623df7c3f boost::asio::detail::scheduler::do_run_one()
(pid=13977) @ 0x7ff623df8a01 boost::asio::detail::scheduler::run()
(pid=13977) @ 0x7ff623df9952 boost::asio::io_context::run()
(pid=14076) @ 0x7f65c59bf08b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14076) @ 0x7f65c5db8c3f boost::asio::detail::scheduler::do_run_one()
(pid=14076) @ 0x7f65c5db9a01 boost::asio::detail::scheduler::run()
(pid=14076) @ 0x7f65c5dba952 boost::asio::io_context::run()
(pid=14076) @ 0x7f65c59da318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=14085) @ 0x7fcf4ad5608b _ZN5boost4asio6detail18completion_handlerIZN3ray10CoreWorker14HandlePushTaskERKNS3_3rpc15PushTaskRequestEPNS5_13PushTaskReplyESt8functionIFvNS3_6StatusESB_IFvvEESE_EEEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
(pid=14085) @ 0x7fcf4b14fc3f boost::asio::detail::scheduler::do_run_one()
(pid=14085) @ 0x7fcf4b150a01 boost::asio::detail::scheduler::run()
(pid=14085) @ 0x7fcf4b151952 boost::asio::io_context::run()
(pid=14085) @ 0x7fcf4ad71318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=13969) @ 0x7fe76e41bc3f boost::asio::detail::scheduler::do_run_one()
(pid=13969) @ 0x7fe76e41ca01 boost::asio::detail::scheduler::run()
(pid=13969) @ 0x7fe76e41d952 boost::asio::io_context::run()
(pid=13969) @ 0x7fe76e03d318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=14049) @ 0x7f47f394ac3f boost::asio::detail::scheduler::do_run_one()
(pid=14049) @ 0x7f47f394ba01 boost::asio::detail::scheduler::run()
(pid=14049) @ 0x7f47f394c952 boost::asio::io_context::run()
(pid=14049) @ 0x7f47f356c318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=14047) @ 0x7f2526941318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=14047) @ 0x7f2526836d46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=14071) @ 0x7ff42c7ef318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=14071) @ 0x7ff42c6e4d46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=13970) @ 0x7fdadb91b318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=13970) @ 0x7fdadb810d46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=13990) @ 0x7f2381113318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=13990) @ 0x7f2381008d46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=14037) @ 0x7f5e32c68318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=14037) @ 0x7f5e32b5dd46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=14033) @ 0x7f102c0b6318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=14033) @ 0x7f102bfabd46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=13977) @ 0x7ff623a19318 ray::CoreWorkerProcess::RunTaskExecutionLoop()
(pid=13977) @ 0x7ff62390ed46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=14076) @ 0x7f65c58cfd46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=14085) @ 0x7fcf4ac66d46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=13969) @ 0x7fe76df32d46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=14049) @ 0x7f47f3461d46 __pyx_pw_3ray_7_raylet_10CoreWorker_5run_task_loop()
(pid=14076) @ 0x555c43897714 method_vectorcall_NOARGS
(pid=14076) @ 0x555c438f46bd _PyEval_EvalFrameDefault
(pid=14076) @ 0x555c438e0387 _PyFunction_Vectorcall
(pid=14076) @ 0x555c438f46bd _PyEval_EvalFrameDefault
(pid=14076) @ 0x555c438df190 _PyEval_EvalCodeWithName
(pid=14076) @ 0x555c438e0a53 PyEval_EvalCode
(pid=14076) @ 0x555c439546d2 run_eval_code_obj
(pid=14076) @ 0x555c43967532 run_mod
(pid=14076) @ 0x555c4396a61e PyRun_FileExFlags
(pid=14076) @ 0x555c4396a809 PyRun_SimpleFileExFlags
(pid=14076) @ 0x555c4396acbe Py_RunMain
(pid=14076) @ 0x555c4396aeb9 Py_BytesMain
(pid=14076) @ 0x7f65c7769b97 __libc_start_main
(pid=14085) @ 0x55fe0f55e714 method_vectorcall_NOARGS
(pid=14085) @ 0x55fe0f5bb6bd _PyEval_EvalFrameDefault
(pid=14085) @ 0x55fe0f5a7387 _PyFunction_Vectorcall
(pid=14085) @ 0x55fe0f5bb6bd _PyEval_EvalFrameDefault
(pid=14085) @ 0x55fe0f5a6190 _PyEval_EvalCodeWithName
(pid=14085) @ 0x55fe0f5a7a53 PyEval_EvalCode
(pid=14085) @ 0x55fe0f61b6d2 run_eval_code_obj
(pid=14085) @ 0x55fe0f62e532 run_mod
(pid=14085) @ 0x55fe0f63161e PyRun_FileExFlags
(pid=14085) @ 0x55fe0f631809 PyRun_SimpleFileExFlags
(pid=14085) @ 0x55fe0f631cbe Py_RunMain
(pid=14085) @ 0x55fe0f631eb9 Py_BytesMain
(pid=14085) @ 0x7fcf4cb00b97 __libc_start_main
(pid=13969) @ 0x561833d06714 method_vectorcall_NOARGS
(pid=13969) @ 0x561833d636bd _PyEval_EvalFrameDefault
(pid=13969) @ 0x561833d4f387 _PyFunction_Vectorcall
(pid=13969) @ 0x561833d636bd _PyEval_EvalFrameDefault
(pid=13969) @ 0x561833d4e190 _PyEval_EvalCodeWithName
(pid=13969) @ 0x561833d4fa53 PyEval_EvalCode
(pid=13969) @ 0x561833dc36d2 run_eval_code_obj
(pid=13969) @ 0x561833dd6532 run_mod
(pid=13969) @ 0x561833dd961e PyRun_FileExFlags
(pid=13969) @ 0x561833dd9809 PyRun_SimpleFileExFlags
(pid=13969) @ 0x561833dd9cbe Py_RunMain
(pid=13969) @ 0x561833dd9eb9 Py_BytesMain
(pid=13969) @ 0x7fe76fdccb97 __libc_start_main
(pid=14049) @ 0x56057a540714 method_vectorcall_NOARGS
(pid=14049) @ 0x56057a59d6bd _PyEval_EvalFrameDefault
(pid=14049) @ 0x56057a589387 _PyFunction_Vectorcall
(pid=14049) @ 0x56057a59d6bd _PyEval_EvalFrameDefault
(pid=14049) @ 0x56057a588190 _PyEval_EvalCodeWithName
(pid=14049) @ 0x56057a589a53 PyEval_EvalCode
(pid=14049) @ 0x56057a5fd6d2 run_eval_code_obj
(pid=14049) @ 0x56057a610532 run_mod
(pid=14049) @ 0x56057a61361e PyRun_FileExFlags
(pid=14049) @ 0x56057a613809 PyRun_SimpleFileExFlags
(pid=14049) @ 0x56057a613cbe Py_RunMain
(pid=14049) @ 0x56057a613eb9 Py_BytesMain
(pid=14049) @ 0x7f47f52fbb97 __libc_start_main
(pid=14047) @ 0x562f96816714 method_vectorcall_NOARGS
(pid=14047) @ 0x562f968736bd _PyEval_EvalFrameDefault
(pid=14047) @ 0x562f9685f387 _PyFunction_Vectorcall
(pid=14047) @ 0x562f968736bd _PyEval_EvalFrameDefault
(pid=14047) @ 0x562f9685e190 _PyEval_EvalCodeWithName
(pid=14047) @ 0x562f9685fa53 PyEval_EvalCode
(pid=14047) @ 0x562f968d36d2 run_eval_code_obj
(pid=14047) @ 0x562f968e6532 run_mod
(pid=14047) @ 0x562f968e961e PyRun_FileExFlags
(pid=14047) @ 0x562f968e9809 PyRun_SimpleFileExFlags
(pid=14047) @ 0x562f968e9cbe Py_RunMain
(pid=14047) @ 0x562f968e9eb9 Py_BytesMain
(pid=14047) @ 0x7f25286d0b97 __libc_start_main
(pid=14071) @ 0x55b73d355714 method_vectorcall_NOARGS
(pid=14071) @ 0x55b73d3b26bd _PyEval_EvalFrameDefault
(pid=14071) @ 0x55b73d39e387 _PyFunction_Vectorcall
(pid=14071) @ 0x55b73d3b26bd _PyEval_EvalFrameDefault
(pid=14071) @ 0x55b73d39d190 _PyEval_EvalCodeWithName
(pid=14071) @ 0x55b73d39ea53 PyEval_EvalCode
(pid=14071) @ 0x55b73d4126d2 run_eval_code_obj
(pid=14071) @ 0x55b73d425532 run_mod
(pid=14071) @ 0x55b73d42861e PyRun_FileExFlags
(pid=14071) @ 0x55b73d428809 PyRun_SimpleFileExFlags
(pid=14071) @ 0x55b73d428cbe Py_RunMain
(pid=14071) @ 0x55b73d428eb9 Py_BytesMain
(pid=14071) @ 0x7ff42e57eb97 __libc_start_main
(pid=13970) @ 0x55d2ba726714 method_vectorcall_NOARGS
(pid=13970) @ 0x55d2ba7836bd _PyEval_EvalFrameDefault
(pid=13970) @ 0x55d2ba76f387 _PyFunction_Vectorcall
(pid=13970) @ 0x55d2ba7836bd _PyEval_EvalFrameDefault
(pid=13970) @ 0x55d2ba76e190 _PyEval_EvalCodeWithName
(pid=13970) @ 0x55d2ba76fa53 PyEval_EvalCode
(pid=13970) @ 0x55d2ba7e36d2 run_eval_code_obj
(pid=13970) @ 0x55d2ba7f6532 run_mod
(pid=13970) @ 0x55d2ba7f961e PyRun_FileExFlags
(pid=13970) @ 0x55d2ba7f9809 PyRun_SimpleFileExFlags
(pid=13970) @ 0x55d2ba7f9cbe Py_RunMain
(pid=13970) @ 0x55d2ba7f9eb9 Py_BytesMain
(pid=13970) @ 0x7fdadd6aab97 __libc_start_main
(pid=13990) @ 0x55915a53b714 method_vectorcall_NOARGS
(pid=13990) @ 0x55915a5986bd _PyEval_EvalFrameDefault
(pid=13990) @ 0x55915a584387 _PyFunction_Vectorcall
(pid=13990) @ 0x55915a5986bd _PyEval_EvalFrameDefault
(pid=13990) @ 0x55915a583190 _PyEval_EvalCodeWithName
(pid=13990) @ 0x55915a584a53 PyEval_EvalCode
(pid=13990) @ 0x55915a5f86d2 run_eval_code_obj
(pid=13990) @ 0x55915a60b532 run_mod
(pid=13990) @ 0x55915a60e61e PyRun_FileExFlags
(pid=13990) @ 0x55915a60e809 PyRun_SimpleFileExFlags
(pid=13990) @ 0x55915a60ecbe Py_RunMain
(pid=13990) @ 0x55915a60eeb9 Py_BytesMain
(pid=13990) @ 0x7f2382ea2b97 __libc_start_main
(pid=14037) @ 0x55a0a90ca714 method_vectorcall_NOARGS
(pid=14037) @ 0x55a0a91276bd _PyEval_EvalFrameDefault
(pid=14037) @ 0x55a0a9113387 _PyFunction_Vectorcall
(pid=14037) @ 0x55a0a91276bd _PyEval_EvalFrameDefault
(pid=14037) @ 0x55a0a9112190 _PyEval_EvalCodeWithName
(pid=14037) @ 0x55a0a9113a53 PyEval_EvalCode
(pid=14037) @ 0x55a0a91876d2 run_eval_code_obj
(pid=14037) @ 0x55a0a919a532 run_mod
(pid=14037) @ 0x55a0a919d61e PyRun_FileExFlags
(pid=14037) @ 0x55a0a919d809 PyRun_SimpleFileExFlags
(pid=14037) @ 0x55a0a919dcbe Py_RunMain
(pid=14037) @ 0x55a0a919deb9 Py_BytesMain
(pid=14037) @ 0x7f5e349f7b97 __libc_start_main
(pid=14033) @ 0x556beb1b9714 method_vectorcall_NOARGS
(pid=14033) @ 0x556beb2166bd _PyEval_EvalFrameDefault
(pid=14033) @ 0x556beb202387 _PyFunction_Vectorcall
(pid=14033) @ 0x556beb2166bd _PyEval_EvalFrameDefault
(pid=14033) @ 0x556beb201190 _PyEval_EvalCodeWithName
(pid=14033) @ 0x556beb202a53 PyEval_EvalCode
(pid=14033) @ 0x556beb2766d2 run_eval_code_obj
(pid=14033) @ 0x556beb289532 run_mod
(pid=14033) @ 0x556beb28c61e PyRun_FileExFlags
(pid=14033) @ 0x556beb28c809 PyRun_SimpleFileExFlags
(pid=14033) @ 0x556beb28ccbe Py_RunMain
(pid=14033) @ 0x556beb28ceb9 Py_BytesMain
(pid=14033) @ 0x7f102de45b97 __libc_start_main
(pid=13977) @ 0x55e5f34b8714 method_vectorcall_NOARGS
(pid=13977) @ 0x55e5f35156bd _PyEval_EvalFrameDefault
(pid=13977) @ 0x55e5f3501387 _PyFunction_Vectorcall
(pid=13977) @ 0x55e5f35156bd _PyEval_EvalFrameDefault
(pid=13977) @ 0x55e5f3500190 _PyEval_EvalCodeWithName
(pid=13977) @ 0x55e5f3501a53 PyEval_EvalCode
(pid=13977) @ 0x55e5f35756d2 run_eval_code_obj
(pid=13977) @ 0x55e5f3588532 run_mod
(pid=13977) @ 0x55e5f358b61e PyRun_FileExFlags
(pid=13977) @ 0x55e5f358b809 PyRun_SimpleFileExFlags
(pid=13977) @ 0x55e5f358bcbe Py_RunMain
(pid=13977) @ 0x55e5f358beb9 Py_BytesMain
(pid=13977) @ 0x7ff6257a8b97 __libc_start_main
E1003 11:42:10.228475 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=b19fee1fe487d333ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.231305 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=88866c7daffdd00effffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.281814 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=7e59d015e24de364ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.448575 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=7f10737098927148ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.452558 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=b9c4a7c684b2e4e7ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.458562 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=bc4ce58d94c1adbeffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.517500 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=b816c6b04d3a3fddffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.547428 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=be3cb809d884a562ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.613471 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=28a1b9e24c223839ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.629298 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=217ee3264e894e29ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.632793 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=bb11ab360805a246ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.644935 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=39c9e7f217500015ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.678879 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=3106d80c4e3c2369ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.679587 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=ad5fa5cbee4768ebffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.679778 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=e39253bc5cc95aacffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.706033 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=8fcb3d859e485b82ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.726506 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=db61220173a4110bffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.770324 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=62223d85d5e7cd76ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.776290 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=b6252bd557d6469effffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.809787 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=b299cffadddbb85affffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.818055 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=bd5c5340e4cc89c2ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.819077 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=ef6d8f9f3dbed41affffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.940109 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=df5a1a828c9685d3ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.963627 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=b7603b6cc81d46c1ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:10.996244 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=3db7cfefbaf00fd0ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.017360 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=87b4f726eda7082bffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.088102 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=b175aa9d8c566fa3ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.173591 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=e4094a5688c1b721ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.178318 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=7bbd90284b71e599ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.250260 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=041697ac73dae98affffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.265493 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=0d247c272666e167ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.271207 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=6a1c8ce65b6afac3ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.328295 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=99506fd459680ea2ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.367713 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=86f1951d408f205fffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.424326 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=d1339b52650f20e3ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
E1003 11:42:11.424485 13780 15370 core_worker.cc:411] Will resubmit task after a 5000ms delay: Type=NORMAL_TASK, Language=PYTHON, Resources: {CPU: 1, }, function_descriptor={type=PythonFunctionDescriptor, module_name=modin.engines.ray.task_wrapper, class_name=, function_name=deploy_ray_func, function_hash=44a6da9f980303781145b4a606abb35e33fa9410}, task_id=0141d1022cd88de1ffffffff01000000, task_name=deploy_ray_func(), job_id=01000000, num_args=4, num_returns=98
---------------------------------------------------------------------------
RayTaskError Traceback (most recent call last)
/databank/jack/projects/code/alltransform.py in <module>
611
612
--> 613 GENE()
614 # META320()
615 # META320_CORESHARED()
/databank/jack/projects/code/alltransform.py in GENE()
221 csvdtype.update({i:np.float16 for i in range(1, 211)})
222 print(len(csvdtype))
--> 223 sheepgenedf = pd.read_csv(sheepgenefile, memory_map=True, nrows=readnrows, dtype=csvdtype)
224 # sheepgenedf.dropna(axis=0, how='all', inplace=True)
225 # sheepgenedf.rename(columns={"Name":"Gene"}, inplace=True)
~/anaconda3/lib/python3.8/site-packages/modin/pandas/io.py in parser_func(filepath_or_buffer, sep, delimiter, header, names, index_col, usecols, squeeze, prefix, mangle_dupe_cols, dtype, engine, converters, true_values, false_values, skipinitialspace, skiprows, nrows, na_values, keep_default_na, na_filter, verbose, skip_blank_lines, parse_dates, infer_datetime_format, keep_date_col, date_parser, dayfirst, cache_dates, iterator, chunksize, compression, thousands, decimal, lineterminator, quotechar, quoting, escapechar, comment, encoding, dialect, error_bad_lines, warn_bad_lines, skipfooter, doublequote, delim_whitespace, low_memory, memory_map, float_precision)
110 if kwargs.get("sep", sep) is False:
111 kwargs["sep"] = "\t"
--> 112 return _read(**kwargs)
113
114 return parser_func
~/anaconda3/lib/python3.8/site-packages/modin/pandas/io.py in _read(**kwargs)
125 from modin.data_management.factories.dispatcher import EngineDispatcher
126
--> 127 pd_obj = EngineDispatcher.read_csv(**kwargs)
128 # This happens when `read_csv` returns a TextFileReader object for iterating through
129 if isinstance(pd_obj, pandas.io.parsers.TextFileReader):
~/anaconda3/lib/python3.8/site-packages/modin/data_management/factories/dispatcher.py in read_csv(cls, **kwargs)
111 @classmethod
112 def read_csv(cls, **kwargs):
--> 113 return cls.__engine._read_csv(**kwargs)
114
115 @classmethod
~/anaconda3/lib/python3.8/site-packages/modin/data_management/factories/factories.py in _read_csv(cls, **kwargs)
85 @classmethod
86 def _read_csv(cls, **kwargs):
---> 87 return cls.io_cls.read_csv(**kwargs)
88
89 @classmethod
~/anaconda3/lib/python3.8/site-packages/modin/engines/base/io/file_reader.py in read(cls, *args, **kwargs)
27 @classmethod
28 def read(cls, *args, **kwargs):
---> 29 query_compiler = cls._read(*args, **kwargs)
30 # TODO (devin-petersohn): Make this section more general for non-pandas kernel
31 # implementations.
~/anaconda3/lib/python3.8/site-packages/modin/engines/base/io/text/csv_reader.py in _read(cls, filepath_or_buffer, **kwargs)
169 # or based on the column(s) that were requested.
170 if index_col is None:
--> 171 row_lengths = cls.materialize(index_ids)
172 new_index = pandas.RangeIndex(sum(row_lengths))
173 # pandas has a really weird edge case here.
~/anaconda3/lib/python3.8/site-packages/modin/engines/ray/task_wrapper.py in materialize(cls, obj_id)
27 @classmethod
28 def materialize(cls, obj_id):
---> 29 return ray.get(obj_id)
~/anaconda3/lib/python3.8/site-packages/ray/worker.py in get(object_refs, timeout)
1426 worker.core_worker.dump_object_store_memory_usage()
1427 if isinstance(value, RayTaskError):
-> 1428 raise value.as_instanceof_cause()
1429 else:
1430 raise value
RayTaskError: ray::deploy_ray_func() (pid=14041, ip=172.29.1.85)
File "python/ray/_raylet.pyx", line 498, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 499, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 1353, in ray._raylet.CoreWorker.store_task_outputs
File "python/ray/_raylet.pyx", line 150, in ray._raylet.check_status
ray.exceptions.RaySystemError: System error: Broken pipe
In [1]: E1003 11:42:16.218492 13780 15370 direct_task_transport.cc:331] The worker failed to receive a response from the local raylet. This is most likely because the local raylet has crahsed.
F1003 11:42:16.218542 13780 15370 direct_task_transport.cc:334] IOError: 14: failed to connect to all addresses
*** Check failure stack trace: ***
@ 0x7f309426b6ed google::LogMessage::Fail()
@ 0x7f309426c84c google::LogMessage::SendToLog()
@ 0x7f309426b3c9 google::LogMessage::Flush()
@ 0x7f309426b5e1 google::LogMessage::~LogMessage()
@ 0x7f3094222789 ray::RayLog::~RayLog()
@ 0x7f3093f01ffc _ZZN3ray29CoreWorkerDirectTaskSubmitter24RequestNewWorkerIfNeededERKSt5tupleIIiSt6vectorINS_8ObjectIDESaIS3_EENS_7ActorIDEEEPKNS_3rpc7AddressEENKUlRKNS_6StatusERKNSA_23RequestWorkerLeaseReplyEE_clESG_SJ_
@ 0x7f3093f40965 ray::rpc::ClientCallImpl<>::OnReplyReceived()
@ 0x7f3093ea0c30 _ZN5boost4asio6detail18completion_handlerIZN3ray3rpc17ClientCallManager29PollEventsFromCompletionQueueEiEUlvE_E11do_completeEPvPNS1_19scheduler_operationERKNS_6system10error_codeEm
@ 0x7f30942ccc3f boost::asio::detail::scheduler::do_run_one()
@ 0x7f30942cda01 boost::asio::detail::scheduler::run()
@ 0x7f30942ce952 boost::asio::io_context::run()
@ 0x7f3093e82310 ray::CoreWorker::RunIOService()
@ 0x7f3094ef7421 execute_native_thread_routine_compat
@ 0x7f30aae7d6db start_thread
@ 0x7f30aaba6a3f clone
Aborted (core dumped)