Add a distributed Dataframe API to Ray #1330

devin-petersohn · 2017-12-16T06:53:40Z

What do these changes do?

Adds a Pandas DataFrame wrapper for use in Ray:

Distributes the DataFrame.
Operates on the DataFrame partitions in remote tasks.
Currently provides some simple functionality.

TODO

Add/Format unit tests in test/dataframe.py
Clean up a couple of quick hacks
~~Add skeleton for full API with NotImplementedErrors~~
~~Docs in doc/source~~
~~Benchmark~~

Push the last 3 to next PR.

AmplabJenkins · 2017-12-16T06:54:05Z

Can one of the admins verify this patch?

devin-petersohn · 2017-12-16T06:54:58Z

I'll ping when the tests are in. Jenkins doesn't need to test until then.

pcmoritz · 2017-12-16T22:49:30Z

ok to test

robertnishihara · 2017-12-16T22:52:13Z

add to whitelist

robertnishihara · 2017-12-16T22:54:53Z

Not a big deal, but the "add to whitelist" command doesn't seem to work for me. I had to go and add @devin-petersohn manually to the jenkins config.

cc @shaneknapp

robertnishihara

Want to add some simple tests for this? E.g., make a file test/dataframe_test.py and call it from .travis.yml.

Note that most of our tests use unittest but I think we'd like to switch to pytest, so if you make a new test file, let's use pytest.

robertnishihara · 2017-12-16T22:58:37Z

python/ray/dataframe.py

+ray.register_custom_serializer(pd.core.indexes.base.Index, use_pickle=True)
+
+
+class DataFrame:


Let's do DataFrame(object) to make sure things work well in Python 2.

robertnishihara · 2017-12-16T22:59:18Z

python/ray/dataframe.py

@@ -0,0 +1,467 @@
+import pandas as pd


Let's include

from __future__ import absolute_import from __future__ import division from __future__ import print_function

at the top of every Python file (to make things consistent across Python 2 and 3).

robertnishihara · 2017-12-16T23:01:00Z

python/ray/dataframe.py

+import numpy as np
+import ray
+
+ray.register_custom_serializer(pd.DataFrame, use_pickle=True)


this is to work around #1305?

Yes, it turns out the pd.core.indexes.base.Index was also giving problems.

robertnishihara · 2017-12-16T23:03:29Z

python/ray/dataframe.py

@@ -0,0 +1,467 @@
+import pandas as pd
+import numpy as np
+import ray


I think it makes sense to move the file to python/ray/dataframe/dataframe.py. And to add a file python/ray/dataframe/__init__.py that has some lines like

from .dataframe import DataFrame

AmplabJenkins · 2017-12-16T23:36:39Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-16T23:36:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2822/
Test PASSed.

devin-petersohn · 2017-12-17T09:16:06Z

.travis.yml

@@ -122,6 +122,7 @@ script:
  - python test/trial_runner_test.py
  - python test/trial_scheduler_test.py
  - python test/cython_test.py
+  - python -m pytest test/dataframe.py


Testing through pytest.

AmplabJenkins · 2017-12-17T09:46:57Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-17T09:46:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2824/
Test PASSed.

robertnishihara · 2017-12-19T00:03:50Z

@devin-petersohn Can you take a look at the Travis test failures at https://travis-ci.org/ray-project/ray/builds/317618386?utm_source=github_status&utm_medium=notification?

Looks like we need to install pandas on Travis. The files to modify are doc/requirements-doc.txt and .travis/install-dependencies.sh.

pcmoritz · 2017-12-19T00:27:28Z

You will also need to solve the rebase conflict, otherwise Travis doesn't run.

AmplabJenkins · 2017-12-19T01:07:03Z

Build finished. Test PASSed.

AmplabJenkins · 2017-12-19T01:07:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2838/
Test PASSed.

AmplabJenkins · 2017-12-19T01:20:31Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-19T01:20:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2840/
Test PASSed.

AmplabJenkins · 2017-12-19T02:39:17Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-19T02:39:17Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2844/
Test PASSed.

AmplabJenkins · 2017-12-19T04:55:21Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-19T04:55:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2846/
Test PASSed.

AmplabJenkins · 2017-12-19T05:59:45Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-19T05:59:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2847/
Test PASSed.

AmplabJenkins · 2017-12-19T22:43:02Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-19T22:43:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2855/
Test PASSed.

robertnishihara · 2017-12-19T23:59:51Z

python/ray/dataframe/dataframe.py

+import ray
+
+ray.register_custom_serializer(pd.DataFrame, use_pickle=True)
+ray.register_custom_serializer(pd.core.indexes.base.Index, use_pickle=True)


now that this is in __init__.py, we can probably remove it from here

robertnishihara · 2017-12-20T00:03:49Z

python/ray/dataframe/__init__.py

@@ -0,0 +1,8 @@
+from .dataframe import DataFrame


from __future__ import absolute_import from __future__ import division from __future__ import print_function

Sorry about that! Please see above.

robertnishihara · 2017-12-20T00:05:23Z

test/dataframe.py

@@ -0,0 +1,177 @@
+import pytest


also add the "from future" stuff here

Added above.

pcmoritz · 2017-12-20T00:58:41Z

There are a few listing errors: https://travis-ci.org/ray-project/ray/jobs/318924011

Do you want to fix the linting and then we get the PR merged and work on the TODOs you mention above in the next PR?

AmplabJenkins · 2017-12-20T01:09:57Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-20T01:09:58Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2856/
Test PASSed.

AmplabJenkins · 2017-12-20T02:23:08Z

Merged build finished. Test PASSed.

AmplabJenkins · 2017-12-20T02:23:08Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/2857/
Test PASSed.

robertnishihara reviewed Dec 16, 2017

View reviewed changes

devin-petersohn commented Dec 17, 2017

View reviewed changes

devin-petersohn added 11 commits December 18, 2017 16:31

Adding dataframe object and minor APIs

abc5376

Adding reduce functionality

1eb5f24

Adding some print and making reduce work on current Ray

9279e41

Cleanup

f13cc38

Added new functionality and docs.

e736d3f

Adding more functionality.

44b5e15

New functionality with older cleanup

9430d5b

Complying with flake8 formatting

d9fa1be

Added tests and addressed reviewer comments

8f1f91b

Complying with flake8.

7de0254

Adding pandas to travis and requirements doc

8f3f054

devin-petersohn force-pushed the dataframe_api branch from 29c40b2 to 8f3f054 Compare December 19, 2017 00:33

Fixing flake8 failures

26a68c3

Fixing flake8 errors from imports

6bf51d6

Fixing import error

7dddd0e

Fixing import errors

23b6222

robertnishihara reviewed Dec 20, 2017

View reviewed changes

Addressing reviewer comments

3b32a84

Addressing lint error

e171e66

pcmoritz changed the title ~~[WIP] Add a distributed Dataframe API to Ray~~ Add a distributed Dataframe API to Ray Dec 20, 2017

pcmoritz self-requested a review December 20, 2017 17:30

pcmoritz approved these changes Dec 20, 2017

View reviewed changes

pcmoritz merged commit a75a473 into ray-project:master Dec 20, 2017

simon-mo mentioned this pull request Jan 31, 2018

[DataFrame] MVP (1/4) #1495

Merged

		ray.register_custom_serializer(pd.core.indexes.base.Index, use_pickle=True)


		class DataFrame:

Add a distributed Dataframe API to Ray #1330

Add a distributed Dataframe API to Ray #1330

Conversation

devin-petersohn commented Dec 16, 2017 • edited Loading

What do these changes do?

TODO

AmplabJenkins commented Dec 16, 2017

devin-petersohn commented Dec 16, 2017

pcmoritz commented Dec 16, 2017

robertnishihara commented Dec 16, 2017

robertnishihara commented Dec 16, 2017

robertnishihara left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Dec 16, 2017

AmplabJenkins commented Dec 16, 2017

Choose a reason for hiding this comment

AmplabJenkins commented Dec 17, 2017

AmplabJenkins commented Dec 17, 2017

robertnishihara commented Dec 19, 2017 • edited Loading

pcmoritz commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

AmplabJenkins commented Dec 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pcmoritz commented Dec 20, 2017

AmplabJenkins commented Dec 20, 2017

AmplabJenkins commented Dec 20, 2017

AmplabJenkins commented Dec 20, 2017

AmplabJenkins commented Dec 20, 2017

devin-petersohn commented Dec 16, 2017 •

edited

Loading

robertnishihara commented Dec 19, 2017 •

edited

Loading