[DataFrame] Implements mode, to_datetime, and get_dummies #1956

kunalgosar · 2018-04-27T08:02:43Z

Import unique and value_counts directly from pandas.

This PR implements mode, to_datetime, and get_dummies and includes a fix for __getitem__.

AmplabJenkins · 2018-04-27T09:05:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5083/
Test PASSed.

AmplabJenkins · 2018-04-27T09:34:09Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5084/
Test PASSed.

AmplabJenkins · 2018-04-27T17:01:25Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5087/
Test FAILed.

devin-petersohn

Left a few comments, overall looks good.

devin-petersohn · 2018-04-27T18:20:01Z

python/ray/dataframe/dataframe.py

+        def mode_helper(df):
+            if numeric_only:
+                df = df.select_dtypes(exclude='object')
+            mode_df = df.mode(axis=axis)


add numeric_only to remove the check.

devin-petersohn · 2018-04-27T18:32:35Z

python/ray/dataframe/datetime.py

+                                  infer_datetime_format=infer_datetime_format,
+                                  origin=origin)
+    if errors == 'raise':
+        pandas.to_datetime(pandas.DataFrame(columns=arg.columns))


For the error checking, do you need all of the other parameters?

devin-petersohn · 2018-04-27T18:41:47Z

python/ray/dataframe/reshape.py

@@ -0,0 +1,106 @@
+import ray


from future

AmplabJenkins · 2018-04-27T19:17:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5089/
Test PASSed.

p-yang · 2018-04-27T21:59:23Z

python/ray/dataframe/dataframe.py

+            df = df.reindex(pd.RangeIndex(max_len), axis=axis)
+            return df
+
+        if axis == 0:


You can probably reuse the code across both if cases, since they're largely the same

p-yang · 2018-04-27T22:00:31Z

python/ray/dataframe/dataframe.py

+            mode_df = df.mode(axis=axis)
+            return mode_df, mode_df.shape[axis]
+
+        def reindex_helper(df, *lengths):


Rename this as it's too similar to utils._reindex_helper

kunalgosar · 2018-04-27T23:07:38Z

Thanks @Veryku @devin-petersohn for reviewing! I addressed the comments above.

AmplabJenkins · 2018-04-28T00:11:57Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5093/
Test PASSed.

p-yang · 2018-04-28T01:23:42Z

python/ray/dataframe/reshape.py

+        DataFrame or one-hot encoded data.
+    """
+    if not isinstance(data, DataFrame):
+        pandas.get_dummies(data, prefix=prefix, prefix_sep=prefix_sep,


Missing a return statement?

oh good catch, thanks!

p-yang · 2018-04-28T01:30:02Z

python/ray/dataframe/reshape.py

+        with_dummies = data.drop(columns_to_encode, axis=1)._col_partitions
+        dropped_columns = data.columns.drop(columns_to_encode)
+
+    def remote_func(df, to_drop, prefix, prefix_sep):


This name is pretty undescriptive

AmplabJenkins · 2018-04-28T02:19:46Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5095/
Test PASSed.

AmplabJenkins · 2018-04-28T02:29:15Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5096/
Test PASSed.

AmplabJenkins · 2018-04-28T05:40:32Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5100/
Test PASSed.

AmplabJenkins · 2018-04-28T05:56:54Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5102/
Test PASSed.

devin-petersohn · 2018-05-02T04:48:26Z

Jenkins, retest this please.

AmplabJenkins · 2018-05-02T05:16:35Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5142/
Test FAILed.

devin-petersohn · 2018-05-02T15:05:09Z

Jenkins, retest this please.

AmplabJenkins · 2018-05-02T16:12:37Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5148/
Test PASSed.

devin-petersohn · 2018-05-03T06:21:10Z

Merged, thanks @kunalgosar

* magic-methods: fmt Fix IndentationError Write magic methods for SampleBatch/PartialRollout Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) [DataFrame] Fix blocking issue on _IndexMetadata passing (ray-project#1965)

* master: (21 commits) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) add pthread linking (ray-project#1986) [DataFrame] Add layer of abstraction to allow OID instantiation (ray-project#1984) ...

* master: (25 commits) [DataFrame] Add direct pandas imports for MVP (ray-project#1960) Make ActorHandles pickleable, also make proper ActorHandle and ActorC… (ray-project#2007) Expand local_dir in Trial init (ray-project#2013) Fixing ascii error for Python2 (ray-project#2009) [DataFrame] Implements df.update (ray-project#1997) [DataFrame] Implements df.as_matrix (ray-project#2001) [DataFrame] Implement quantile (ray-project#1992) [DataFrame] Impement sort_values and sort_index (ray-project#1977) [DataFrame] Implement rank (ray-project#1991) [DataFrame] Implemented prod, product, added test suite (ray-project#1994) [DataFrame] Implemented __setitem__, select_dtypes, and astype (ray-project#1941) [DataFrame] Implement diff (ray-project#1996) [DataFrame] Implemented nunique, skew (ray-project#1995) [DataFrame] Implements filter and dropna (ray-project#1959) [DataFrame] Implements df.pipe (ray-project#1999) [DataFrame] Apply() for Lists and Dicts (ray-project#1973) Clean up syntax for supported Python versions. (ray-project#1963) [DataFrame] Implements mode, to_datetime, and get_dummies (ray-project#1956) [DataFrame] Fix dtypes (ray-project#1930) keep_dims -> keepdims (ray-project#1980) ...

devin-petersohn reviewed Apr 27, 2018

View reviewed changes

p-yang reviewed Apr 27, 2018

View reviewed changes

p-yang reviewed Apr 28, 2018

View reviewed changes

kunalgosar force-pushed the mode branch from b6fb3c6 to 2243b8a Compare April 28, 2018 04:38

kunalgosar added 14 commits May 1, 2018 00:17

implement mode and fix getitem

92c1ac5

mode broken on misaligned partitions

5c017f0

fully implement mode

d69d60b

implement to_datetime

61e16b3

implement get_dummies

9c8f8a9

implement tests

a621d8f

fix __getitem__

430360e

fix python2 compatibility

40f66ea

fix getitem bug

767dc8f

resolving comments

7f89d9f

Adding documentation

102e27d

resolving comment

bc2a443

resolve name change

5a0b4b5

speeding up getitem

928aed3

kunalgosar force-pushed the mode branch from 8001718 to 928aed3 Compare May 1, 2018 07:24

complete rebase

44d0a0d

devin-petersohn approved these changes May 2, 2018

View reviewed changes

devin-petersohn merged commit d85ee0b into ray-project:master May 3, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DataFrame] Implements mode, to_datetime, and get_dummies #1956

[DataFrame] Implements mode, to_datetime, and get_dummies #1956

kunalgosar commented Apr 27, 2018

AmplabJenkins commented Apr 27, 2018

AmplabJenkins commented Apr 27, 2018

AmplabJenkins commented Apr 27, 2018

devin-petersohn left a comment

devin-petersohn Apr 27, 2018

kunalgosar Apr 27, 2018

devin-petersohn Apr 27, 2018

kunalgosar Apr 27, 2018

devin-petersohn Apr 27, 2018

kunalgosar Apr 27, 2018

AmplabJenkins commented Apr 27, 2018

p-yang Apr 27, 2018

kunalgosar Apr 27, 2018

p-yang Apr 27, 2018 •

edited

Loading

kunalgosar Apr 27, 2018

kunalgosar commented Apr 27, 2018

AmplabJenkins commented Apr 28, 2018

p-yang Apr 28, 2018

kunalgosar Apr 28, 2018

p-yang Apr 28, 2018

kunalgosar Apr 28, 2018

AmplabJenkins commented Apr 28, 2018

AmplabJenkins commented Apr 28, 2018

AmplabJenkins commented Apr 28, 2018

AmplabJenkins commented Apr 28, 2018

devin-petersohn commented May 2, 2018

AmplabJenkins commented May 2, 2018

devin-petersohn commented May 2, 2018

AmplabJenkins commented May 2, 2018

devin-petersohn commented May 3, 2018

[DataFrame] Implements mode, to_datetime, and get_dummies #1956

[DataFrame] Implements mode, to_datetime, and get_dummies #1956

Conversation

kunalgosar commented Apr 27, 2018

AmplabJenkins commented Apr 27, 2018

AmplabJenkins commented Apr 27, 2018

AmplabJenkins commented Apr 27, 2018

devin-petersohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Apr 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

p-yang Apr 27, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kunalgosar commented Apr 27, 2018

AmplabJenkins commented Apr 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Apr 28, 2018

AmplabJenkins commented Apr 28, 2018

AmplabJenkins commented Apr 28, 2018

AmplabJenkins commented Apr 28, 2018

devin-petersohn commented May 2, 2018

AmplabJenkins commented May 2, 2018

devin-petersohn commented May 2, 2018

AmplabJenkins commented May 2, 2018

devin-petersohn commented May 3, 2018

p-yang Apr 27, 2018 •

edited

Loading