Ray fails to serialize Torch tensor.

[robertnishihara]

I'm running into problems serializing pytorch tensors.

import ray
import torch

ray.init()

x = torch.Tensor(10)

x_id = ray.put(x)

result = ray.get(x_id)

Inspecting the values shows that x and result are different.

>>> x
 0.0000e+00
-8.5899e+09
 0.0000e+00
-8.5899e+09
 5.6052e-45
 1.4714e-43
 1.5975e-43
 1.4153e-43
 1.3873e-43
 1.6255e-43
[torch.FloatTensor of size 10]

>>> result
[torch.FloatTensor with no dimension]

>>> result[0]
RuntimeError: dimension 1 out of range of 0D tensor at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/TH/generic/THTensor.c:24

cc @vitchyr

[robertnishihara]

The fix for this is waiting on #1138.

[robertnishihara]

No longer waiting on #1138 because it is fixed in pyarrow, so #1152 fixes it.

[richardliaw]

I'm building off master and the above example no longer works.

I'm running python 3.6 on a macOS.

In [1]: import ray
   ...: import torch
   ...:
   ...: ray.init()
   ...:
   ...: x = torch.Tensor(10)
   ...:
   ...: x_id = ray.put(x)
   ...:
   ...: result = ray.get(x_id)
   ...:
Process STDOUT and STDERR is being redirected to /tmp/raylogs/.
Waiting for redis server at 127.0.0.1:40869 to respond...
Waiting for redis server at 127.0.0.1:10446 to respond...
Starting local scheduler with the following resources: {'CPU': 8, 'GPU': 0}.

======================================================================
View the web UI at http://localhost:8889/notebooks/ray_ui8885.ipynb?token=5148456270b22c7a7ca145ddb67789feb86c0d935350b766
======================================================================

WARNING: Serializing objects of type <class 'torch.Tensor'> by expanding them as dictionaries of their fields. This behavior may be incorrect in some cases.

In [2]: result
Out[2]: tensor([])

[richardliaw]

This is blocking #2036.

[robertnishihara]

Does it work with earlier versions of pytorch? Could it be the version update (I assume this is pytorch 0.4)?

It's possible that simply switching to the latest Arrow will fix the issue because of apache/arrow@df9bdc1.

[richardliaw]

sure; how do I try that out?

cc @alok

[alok]

@richardliaw If you use the code snippet I put in #1856 and the version of arrow Robert mentioned (or manually change your local copy of the arrow code since it's a one line fix), it should work. I just tested your code snippet in this issue with the custom serializer and it works fine.

[richardliaw]

Is it possible to get away without custom serializers?

[alok]

Only if you use that arrow PR. I use the custom serializer for local testing and the arrow PR just makes it the official solution.

[robertnishihara]

@richardliaw the line to change is

ray/thirdparty/scripts/build_arrow.sh

Line 50 in 78bb887

git checkout 5f9cf9c96709f92e9ac4828cf3e106a165576ce7