[DataFrame] Implemented setitem, select_dtypes, and astype #1941

jaeminkim1324 · 2018-04-23T22:13:30Z

What do these changes do?

Adds functionality for astype and select_dtypes methods for Dataframes

AmplabJenkins · 2018-04-23T23:17:43Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5041/
Test PASSed.

AmplabJenkins · 2018-04-24T20:35:29Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5052/
Test PASSed.

AmplabJenkins · 2018-04-24T21:47:01Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5053/
Test PASSed.

AmplabJenkins · 2018-04-24T21:49:23Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5054/
Test PASSed.

kunalgosar · 2018-04-25T01:54:18Z

python/ray/dataframe/dataframe.py

-            "github.com/ray-project/ray.")
+        if not columns:
+            return DataFrame()
+        col_idx = [self.columns.get_loc(columns[i])


This needs to handle cases where some of the passed in values are not in self.columns. Pandas fills this case with NaNs.

kunalgosar · 2018-04-25T01:56:51Z

python/ray/dataframe/dataframe.py

@@ -3260,7 +3332,7 @@ def _getitem_array(self, key):
            new_parts = _map_partitions(lambda df: df[key],
                                        self._col_partitions)
            columns = self.columns
-            index = self.index[key]
+            index = self._col_metadata[key].index


Why did this change?

kunalgosar · 2018-04-25T01:58:56Z

python/ray/dataframe/dataframe.py

-        raise NotImplementedError(
-            "To contribute to Pandas on Ray, please visit "
-            "github.com/ray-project/ray.")
+        if errors == 'raise':


Should not ignore errors when errors == 'raise'

@kunalgosar What do you mean by your comment?

On errors == 'raise', these exceptions should be raised. On errors == 'ignore', they should be suppressed and self should be returned as per the pandas documentation. Currently errors are suppressed in this if case.

p-yang · 2018-04-25T03:16:01Z

python/ray/dataframe/dataframe.py

-            "github.com/ray-project/ray.")
+        if errors == 'raise':
+            try:
+                pd.DataFrame().astype(dtype)


This check doesn't work on empty dataframes

p-yang · 2018-04-25T03:25:11Z

python/ray/dataframe/dataframe.py

-        raise NotImplementedError(
-            "To contribute to Pandas on Ray, please visit "
-            "github.com/ray-project/ray.")
+        if not columns:


Anytime that columns == None this will return an empty dataframe

p-yang · 2018-04-25T03:37:54Z

python/ray/dataframe/dataframe.py

+        elif not exclude:
+            exclude = []
+
+        sel = tuple(map(set, (include, exclude)))


This code (and below) is somewhat hard to read, and can be wrapped up into the if-else cases above

The code using map in particular?

p-yang · 2018-04-25T03:41:29Z

python/ray/dataframe/dataframe.py

+
+        axis = pd.DataFrame()._get_axis_number(axis) if (axis) else 0
+        if axis == 1 or columns:
+            def row_helper(df, col_idx):


Since the partitions currently have RangeIndex on both rows and columns, this function won't work. Refer to utils._reindex_helper and ask @devin-petersohn for details.

So I tried to get around that by translating the columns passed in to the helper by using the loc index of the columns desired (col_idx)

Removed this implementation of reindex for a future PR

AmplabJenkins · 2018-04-25T04:35:58Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5062/
Test FAILed.

AmplabJenkins · 2018-04-25T05:02:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5061/
Test PASSed.

AmplabJenkins · 2018-04-25T08:40:30Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5071/
Test PASSed.

devin-petersohn · 2018-05-01T22:54:07Z

python/ray/dataframe/test/test_dataframe.py

@@ -2150,7 +2168,7 @@ def test_reindex():
    ray_df = create_test_dataframe()

    with pytest.raises(NotImplementedError):
-        ray_df.reindex()
+        ray_df.reindex(None)


Please add a sanity check test here.

Looking through the code, was reindex dropped from this PR?

yes, reindex did not have full functionality (axes, errors) so I took it out for the PR against master

kunalgosar · 2018-05-05T01:07:46Z

python/ray/dataframe/test/test_dataframe.py

@@ -2150,7 +2168,7 @@ def test_reindex():
    ray_df = create_test_dataframe()

    with pytest.raises(NotImplementedError):
-        ray_df.reindex()
+        ray_df.reindex(None)


Revert this line.

AmplabJenkins · 2018-05-05T01:29:44Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5229/
Test PASSed.

AmplabJenkins · 2018-05-05T04:42:05Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5230/
Test PASSed.

kunalgosar

Looks great!

devin-petersohn · 2018-05-06T03:27:34Z

Merged, thanks @jaeminkim1324!

* master: (21 commits) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) ...

* master: (25 commits) [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) ...

* master: [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973)

added reindex, __setitem__, select_dtypes, and astype functionality

057f4b1

jaeminkim1324 changed the title ~~added reindex, __setitem__, select_dtypes, and astype functionality~~ [DataFrame] added reindex, __setitem__, select_dtypes, and astype functionality Apr 23, 2018

jaeminkim1324 changed the title ~~[DataFrame] added reindex, __setitem__, select_dtypes, and astype functionality~~ [DataFrame] Added reindex, __setitem__, select_dtypes, and astype functionality Apr 23, 2018

readded tests for astype and select_dtypes

b6da366

jaeminkim1324 added 2 commits April 24, 2018 13:43

fixed index issue with reindex

42daa2d

lint spacing

4a9fa6b

kunalgosar suggested changes Apr 25, 2018

View reviewed changes

p-yang suggested changes Apr 25, 2018

View reviewed changes

jaeminkim1324 added 2 commits April 24, 2018 21:02

removing current reindex implementation for future pr

c2a452d

wrong testing func

081d9dd

errors now raised in the workers, but suppressing them can be an issue

f407529

devin-petersohn reviewed May 1, 2018

View reviewed changes

jaeminkim1324 changed the title ~~[DataFrame] Added reindex, __setitem__, select_dtypes, and astype functionality~~ [DataFrame] Added __setitem__, select_dtypes, and astype functionality May 4, 2018

jaeminkim1324 changed the title ~~[DataFrame] Added __setitem__, select_dtypes, and astype functionality~~ [DataFrame] Implemented __setitem__, select_dtypes, and astype May 4, 2018

updated code for select_dtypes

993582b

kunalgosar suggested changes May 5, 2018

View reviewed changes

Update test_dataframe.py

8ad6e5b

kunalgosar approved these changes May 5, 2018

View reviewed changes

devin-petersohn approved these changes May 6, 2018

View reviewed changes

devin-petersohn merged commit 36fd648 into ray-project:master May 6, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DataFrame] Implemented setitem, select_dtypes, and astype #1941

[DataFrame] Implemented setitem, select_dtypes, and astype #1941

jaeminkim1324 commented Apr 23, 2018

AmplabJenkins commented Apr 23, 2018

AmplabJenkins commented Apr 24, 2018

AmplabJenkins commented Apr 24, 2018

AmplabJenkins commented Apr 24, 2018

kunalgosar Apr 25, 2018

kunalgosar Apr 25, 2018 •

edited

Loading

kunalgosar Apr 25, 2018

devin-petersohn Apr 25, 2018

kunalgosar Apr 25, 2018 •

edited

Loading

p-yang Apr 25, 2018

p-yang Apr 25, 2018

p-yang Apr 25, 2018

jaeminkim1324 Apr 25, 2018

p-yang Apr 25, 2018

jaeminkim1324 Apr 25, 2018

jaeminkim1324 Apr 25, 2018

AmplabJenkins commented Apr 25, 2018

AmplabJenkins commented Apr 25, 2018

AmplabJenkins commented Apr 25, 2018

devin-petersohn May 1, 2018

devin-petersohn May 2, 2018

jaeminkim1324 May 3, 2018

kunalgosar May 5, 2018

AmplabJenkins commented May 5, 2018

AmplabJenkins commented May 5, 2018

kunalgosar left a comment

devin-petersohn commented May 6, 2018

[DataFrame] Implemented __setitem__, select_dtypes, and astype #1941

[DataFrame] Implemented __setitem__, select_dtypes, and astype #1941

Conversation

jaeminkim1324 commented Apr 23, 2018

What do these changes do?

AmplabJenkins commented Apr 23, 2018

AmplabJenkins commented Apr 24, 2018

AmplabJenkins commented Apr 24, 2018

AmplabJenkins commented Apr 24, 2018

Choose a reason for hiding this comment

kunalgosar Apr 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kunalgosar Apr 25, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Apr 25, 2018

AmplabJenkins commented Apr 25, 2018

AmplabJenkins commented Apr 25, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented May 5, 2018

AmplabJenkins commented May 5, 2018

kunalgosar left a comment

Choose a reason for hiding this comment

devin-petersohn commented May 6, 2018

[DataFrame] Implemented setitem, select_dtypes, and astype #1941

[DataFrame] Implemented setitem, select_dtypes, and astype #1941

kunalgosar Apr 25, 2018 •

edited

Loading

kunalgosar Apr 25, 2018 •

edited

Loading