[Core] Please provide better message where 'RuntimeError: Failed to unpickle serialized exception'

[Joshuaalbert]

What happened + What you expected to happen

You could at least tell us what line it's occuring on, and also likely where the origin of the exception is.

(DataStreamer pid=974272) Failed to unpickle serialized exception
(DataStreamer pid=974272) Traceback (most recent call last):
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/exceptions.py", line 51, in from_ray_exception
(DataStreamer pid=974272)     return pickle.loads(ray_exception.serialized_exception)
(DataStreamer pid=974272)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/jax/_src/errors.py", line 314, in __init__
(DataStreamer pid=974272)     f"{tracer._error_repr()}{tracer._origin_msg()}")
(DataStreamer pid=974272)        ^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272) AttributeError: 'str' object has no attribute '_error_repr'
(DataStreamer pid=974272) 
(DataStreamer pid=974272) The above exception was the direct cause of the following exception:
(DataStreamer pid=974272) 
(DataStreamer pid=974272) Traceback (most recent call last):
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/_private/serialization.py", line 460, in deserialize_objects
(DataStreamer pid=974272)     obj = self._deserialize_object(data, metadata, object_ref)
(DataStreamer pid=974272)           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/_private/serialization.py", line 342, in _deserialize_object
(DataStreamer pid=974272)     return RayError.from_bytes(obj)
(DataStreamer pid=974272)            ^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/exceptions.py", line 45, in from_bytes
(DataStreamer pid=974272)     return RayError.from_ray_exception(ray_exception)
(DataStreamer pid=974272)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(DataStreamer pid=974272)   File "/usr/local/lib/python3.11/site-packages/ray/exceptions.py", line 54, in from_ray_exception
(DataStreamer pid=974272)     raise RuntimeError(msg) from e
(DataStreamer pid=974272) RuntimeError: Failed to unpickle serialized exception

Versions / Dependencies

2.39

Reproduction script

Not possible because the poor exception messages makes it hard to find out where the problem is.

Issue Severity

High: It blocks me from completing my task.

[wingkitlee0]

I saw some jax code there. How are you calling the remote function and jax array?

[Joshuaalbert]

@wingkitlee0 I am able to find the source of the problem after a trial and error search with debug messages since JAX is used in many places. However, I'm loath to discuss the cause of the exception (which is simple to solve once found) because I've been asking Ray to put better messages in the case of unserialisable exceptions for some time, and their engineers like to ignore the request and talk rather about the exception. It's an edge case that an exception is not serialisable, and they should handle it.

[celestinoxp]

I have a lot of time a problem, in my case with catboost:

`2025-01-20 05:45:59,271 ERROR serialization.py:462 -- Failed to unpickle serialized exception
Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51, in from_ray_exception
return pickle.loads(ray_exception.serialized_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 460, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 342, in _deserialize_object
return RayError.from_bytes(obj)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45, in from_bytes
return RayError.from_ray_exception(ray_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54, in from_ray_exception
raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
Warning: Exception caused CatBoost_BAG_L1 to fail during training... Skipping this model.
System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51, in from_ray_exception
return pickle.loads(ray_exception.serialized_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 460, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 342, in _deserialize_object
return RayError.from_bytes(obj)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45, in from_bytes
return RayError.from_ray_exception(ray_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54, in from_ray_exception
raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception

Detailed Traceback:
Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\trainer\abstract_trainer.py", line 2106, in _train_and_save
model = self._train_single(**model_fit_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\trainer\abstract_trainer.py", line 1993, in _train_single
model = model.fit(X=X, y=y, X_val=X_val, y_val=y_val, X_test=X_test, y_test=y_test, total_resources=total_resources, **model_fit_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\abstract\abstract_model.py", line 925, in fit
out = self._fit(**kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\stacker_ensemble_model.py", line 270, in _fit
return super()._fit(X=X, y=y, time_limit=time_limit, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\bagged_ensemble_model.py", line 298, in _fit
self._fit_folds(
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\bagged_ensemble_model.py", line 724, in _fit_folds
fold_fitting_strategy.after_all_folds_scheduled()
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 688, in after_all_folds_scheduled
self._run_pseudo_sequential(X, y, X_pseudo, y_pseudo, model_base_ref, time_limit_fold, head_node_id)
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 666, in _run_pseudo_sequential
self._process_fold_results(finished[0], unfinished, fold_ctx)
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 587, in _process_fold_results
raise processed_exception
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 550, in _process_fold_results
fold_model, pred_proba, time_start_fit, time_end_fit, predict_time, predict_1_time, predict_n_size, fit_num_cpus, fit_num_gpus = self.ray.get(finished)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\worker.py", line 2755, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\worker.py", line 908, in get_objects
raise value
ray.exceptions.RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51, in from_ray_exception
return pickle.loads(ray_exception.serialized_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 460, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 342, in _deserialize_object
return RayError.from_bytes(obj)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45, in from_bytes
return RayError.from_ray_exception(ray_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54, in from_ray_exception
raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception

Fitting model: XGBoost_BAG_L1 ... Training model for up to 11962.12s of the 11962.10s of remaining time.
Warning: Potentially not enough memory to safely train model. Estimated to require 6.870 GB out of 8.863 GB available memory (77.514%)... (100.000% of avail memory is the max safe size)
To avoid this warning, specify the model hyperparameter "ag.max_memory_usage_ratio" to a larger value (currently 1.0, set to >=1.08 to avoid the warning)
To set the same value for all models, do the following when calling predictor.fit: predictor.fit(..., ag_args_fit={"ag.max_memory_usage_ratio": VALUE})
Setting "ag.max_memory_usage_ratio" to values above 1 may result in out-of-memory errors. You may consider using a machine with more memory as a safer alternative.
Memory not enough to fit 8 folds in parallel. Will train 1 folds in parallel instead (Estimated 77.51% memory usage per fold, 77.51%/80.00% total).
Fitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy (1 workers, per: cpus=8, gpus=0, memory=77.51%)
Switching to pseudo sequential ParallelFoldFittingStrategy to avoid Python memory leakage.
Overrule this behavior by setting fold_fitting_strategy to 'sequential_local' in ag_args_ensemble when when calling predictor.fit
0.7713 = Validation score (accuracy)
5032.22s = Training runtime
1.52s = Validation runtime
Fitting model: NeuralNetTorch_BAG_L1 ... Training model for up to 6898.29s of the 6898.27s of remaining time.
Memory not enough to fit 8 folds in parallel. Will train 4 folds in parallel instead (Estimated 17.94% memory usage per fold, 71.76%/80.00% total).
Fitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy (4 workers, per: cpus=4, gpus=0, memory=17.94%)
0.6774 = Validation score (accuracy)
414.77s = Training runtime
57.83s = Validation runtime
Fitting model: LightGBMLarge_BAG_L1 ... Training model for up to 6444.74s of the 6444.72s of remaining time.
Warning: Not enough memory to safely train model. Estimated to require 10.857 GB out of 8.934 GB available memory (121.516%)... (90.000% of avail memory is the max safe size)
To force training the model, specify the model hyperparameter "ag.max_memory_usage_ratio" to a larger value (currently 1.0, set to >=1.40 to avoid the error)
To set the same value for all models, do the following when calling predictor.fit: predictor.fit(..., ag_args_fit={"ag.max_memory_usage_ratio": VALUE})
Setting "ag.max_memory_usage_ratio" to values above 1 may result in out-of-memory errors. You may consider using a machine with more memory as a safer alternative.
Not enough memory to train LightGBMLarge_BAG_L1... Skipping this model.
Fitting model: CatBoost_r177_BAG_L1 ... Training model for up to 6415.10s of the 6415.08s of remaining time.
Memory not enough to fit 8 folds in parallel. Will train 1 folds in parallel instead (Estimated 70.50% memory usage per fold, 70.50%/80.00% total).
Fitting 8 child models (S1F1 - S1F8) | Fitting with ParallelLocalFoldFittingStrategy (1 workers, per: cpus=8, gpus=0, memory=70.50%)
Switching to pseudo sequential ParallelFoldFittingStrategy to avoid Python memory leakage.
Overrule this behavior by setting fold_fitting_strategy to 'sequential_local' in ag_args_ensemble when when calling predictor.fit
2025-01-20 07:19:08,105 ERROR serialization.py:462 -- Failed to unpickle serialized exception
Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51, in from_ray_exception
return pickle.loads(ray_exception.serialized_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 460, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 342, in _deserialize_object
return RayError.from_bytes(obj)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45, in from_bytes
return RayError.from_ray_exception(ray_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54, in from_ray_exception
raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception
Warning: Exception caused CatBoost_r177_BAG_L1 to fail during training... Skipping this model.
System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51, in from_ray_exception
return pickle.loads(ray_exception.serialized_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 460, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 342, in _deserialize_object
return RayError.from_bytes(obj)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45, in from_bytes
return RayError.from_ray_exception(ray_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54, in from_ray_exception
raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception

Detailed Traceback:
Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\trainer\abstract_trainer.py", line 2106, in _train_and_save
model = self._train_single(**model_fit_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\trainer\abstract_trainer.py", line 1993, in _train_single
model = model.fit(X=X, y=y, X_val=X_val, y_val=y_val, X_test=X_test, y_test=y_test, total_resources=total_resources, **model_fit_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\abstract\abstract_model.py", line 925, in fit
out = self._fit(**kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\stacker_ensemble_model.py", line 270, in _fit
return super()._fit(X=X, y=y, time_limit=time_limit, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\bagged_ensemble_model.py", line 298, in _fit
self._fit_folds(
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\bagged_ensemble_model.py", line 724, in _fit_folds
fold_fitting_strategy.after_all_folds_scheduled()
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 688, in after_all_folds_scheduled
self._run_pseudo_sequential(X, y, X_pseudo, y_pseudo, model_base_ref, time_limit_fold, head_node_id)
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 666, in _run_pseudo_sequential
self._process_fold_results(finished[0], unfinished, fold_ctx)
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 587, in _process_fold_results
raise processed_exception
File "C:\Users\celes\anaconda3\Lib\site-packages\autogluon\core\models\ensemble\fold_fitting_strategy.py", line 550, in _process_fold_results
fold_model, pred_proba, time_start_fit, time_end_fit, predict_time, predict_1_time, predict_n_size, fit_num_cpus, fit_num_gpus = self.ray.get(finished)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\auto_init_hook.py", line 21, in auto_init_wrapper
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\client_mode_hook.py", line 103, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\worker.py", line 2755, in get
values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\worker.py", line 908, in get_objects
raise value
ray.exceptions.RaySystemError: System error: Failed to unpickle serialized exception
traceback: Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 51, in from_ray_exception
return pickle.loads(ray_exception.serialized_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named '_catboost'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 460, in deserialize_objects
obj = self._deserialize_object(data, metadata, object_ref)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray_private\serialization.py", line 342, in _deserialize_object
return RayError.from_bytes(obj)
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 45, in from_bytes
return RayError.from_ray_exception(ray_exception)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\celes\anaconda3\Lib\site-packages\ray\exceptions.py", line 54, in from_ray_exception
raise RuntimeError(msg) from e
RuntimeError: Failed to unpickle serialized exception`

[teddy309]

How Can I solve this problem. I'm on same case too.