[DataFrame] Handling zero length partitions #2210

kunalgosar · 2018-06-08T01:28:04Z

This filters out zero length partitions on getting col_partitions or row_partitions.

AmplabJenkins · 2018-06-08T01:57:50Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5930/
Test FAILed.

AmplabJenkins · 2018-06-11T07:53:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6001/
Test FAILed.

pschafhalter · 2018-06-11T17:54:33Z

python/ray/dataframe/dataframe.py

+        empty_cols_mask = self._col_metadata._lengths > 0
+        if any(empty_cols_mask):
+            self._col_metadata._lengths \
+                = self._col_metadata._lengths[empty_cols_mask]


Formatting is a bit weird. Move = to previous line to be consistent with line 200?

pschafhalter · 2018-06-11T21:24:28Z

python/ray/dataframe/groupby.py

@@ -208,6 +209,9 @@ def apply_helper(df):
                    num_return_vals=len(new_df._block_partitions))
                    for block in new_df._block_partitions.T]).T
                new_df.index = self._index
+                new_df._row_metadata = \
+                    _IndexMetadata(new_df._block_partitions[:, 0],


The concat from earlier constructs a new DataFrame with a new _IndexMetadata which gets overwritten here. Ideally, we could modify concat to pass this _indexMetadata to the DataFrame constructor, but I'm not sure if this is possible to do elegantly with the current code base.

No need to make a change to the PR, but would like to point this out.

The issue here is that the reindex that is applied on columns changes the partition structure and the old _IndexMetadata created on concat is invalid. I agree that duplicate work is being done here, but I'm not sure there is a nice solution.

I'm planning to rewrite the majority of this class sometime this week, both to make it more performant and more clean.

AmplabJenkins · 2018-06-11T22:43:48Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6009/
Test FAILed.

kunalgosar · 2018-06-12T06:47:54Z

Jenkins, retest this please

AmplabJenkins · 2018-06-12T07:23:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6020/
Test FAILed.

kunalgosar · 2018-06-12T18:14:47Z

Jenkins, retest this please

AmplabJenkins · 2018-06-12T18:48:13Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6030/
Test FAILed.

kunalgosar · 2018-06-12T20:03:05Z

Jenkins, retest this please

AmplabJenkins · 2018-06-12T20:33:07Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6034/
Test FAILed.

kunalgosar · 2018-06-13T00:19:05Z

Jenkins, retest this please

devin-petersohn · 2018-06-18T14:30:15Z

Jenkins, retest this please.

AmplabJenkins · 2018-06-18T15:39:22Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6098/
Test PASSed.

devin-petersohn

Left a few comment. Thanks for the fix!

devin-petersohn · 2018-06-18T16:07:43Z

python/ray/dataframe/groupby.py

@@ -208,6 +209,9 @@ def apply_helper(df):
                    num_return_vals=len(new_df._block_partitions))
                    for block in new_df._block_partitions.T]).T
                new_df.index = self._index
+                new_df._row_metadata = \
+                    _IndexMetadata(new_df._block_partitions[:, 0],


I'm planning to rewrite the majority of this class sometime this week, both to make it more performant and more clean.

devin-petersohn · 2018-06-18T16:17:25Z

python/ray/dataframe/dataframe.py

-        all_empty = ray.get(_map_partitions(
-            lambda df: df.empty, self._row_partitions))
-        return False not in all_empty
+        return self._row_metadata._empty


A more efficient approach would be return len(self.columns) == 0 or len(self.rows) == 0. This would also simplify a lot of the code in _IndexMetadata.

devin-petersohn · 2018-06-18T16:25:48Z

python/ray/dataframe/index_metadata.py

@@ -48,6 +49,9 @@ def __init__(self, dfs=None, index=None, axis=0, lengths_oid=None,
            else:
                lengths_oid = _build_col_widths.remote(dfs)
            coord_df_oid = _build_coord_df.remote(lengths_oid, index)
+            self._empty = _check_empty.remote(dfs)


We are doing this twice, once for each of column_metadata and row_metadata.

devin-petersohn · 2018-06-18T16:31:29Z

python/ray/dataframe/utils.py

+@ray.remote
+def _check_empty(dfs):
+    """Check if all partitions are empty"""
+    return all(ray.get([_deploy_func.remote(_get_empty, d) for d in dfs]))


I don't think you need this (or _get_empty if you make some of the other changes, but a more efficient way to implement something like this is:

return next((False for d in dfs if not ray.get(_deploy_func.remote(_get_empty, d))), True)

AmplabJenkins · 2018-06-20T20:52:53Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6159/
Test FAILed.

kunalgosar · 2018-06-20T21:14:35Z

Jenkins, retest this please.

devin-petersohn · 2018-06-20T21:37:40Z

Jenkins, retest this please.

AmplabJenkins · 2018-06-20T22:41:40Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/6161/
Test PASSed.

* 'master' of https://github.com/ray-project/ray: (157 commits) Fix build failure while using make -j1. Issue 2257 (ray-project#2279) Cast locator with index type (ray-project#2274) fixing zero length partitions (ray-project#2237) Make actor handles work in Python mode. (ray-project#2283) [xray] Add error table and push error messages to driver through node manager. (ray-project#2256) addressing comments (ray-project#2210) Re-enable some actor tests. (ray-project#2276) Experimental: enable automatic GCS flushing with configurable policy. (ray-project#2266) [xray] Sets good object manager defaults. (ray-project#2255) [tune] Update Trainable doc to expose interface (ray-project#2272) [rllib] Add a simple REST policy server and client example (ray-project#2232) [asv] Pushing to s3 (ray-project#2246) [rllib] Remove need to pass around registry (ray-project#2250) Support multiple availability zones in AWS (fix ray-project#2177) (ray-project#2254) [rllib] Add squash_to_range model option (ray-project#2239) Mitigate randomly building failure: adding gen_local_scheduler_fbs to raylet lib. (ray-project#2271) [rllib] Refactor Multi-GPU for PPO (ray-project#1646) [rllib] Envs for vectorized execution, async execution, and policy serving (ray-project#2170) [Dataframe] Change pandas and ray.dataframe imports (ray-project#1942) [Java] Replace binary rewrite with Remote Lambda Cache (SerdeLambda) (ray-project#2245) ...

kunalgosar force-pushed the zero_length_partitions branch from b970a4c to 1da7e03 Compare June 11, 2018 06:47

kunalgosar changed the title ~~[WIP] [DataFrame] Handling zero length partitions~~ [DataFrame] Handling zero length partitions Jun 11, 2018

pschafhalter suggested changes Jun 11, 2018

View reviewed changes

kunalgosar force-pushed the zero_length_partitions branch from 1da7e03 to e30c4af Compare June 11, 2018 21:34

pschafhalter approved these changes Jun 11, 2018

View reviewed changes

devin-petersohn requested changes Jun 19, 2018

View reviewed changes

addressing comments

5ccca59

kunalgosar force-pushed the zero_length_partitions branch from e30c4af to 5ccca59 Compare June 20, 2018 20:20

devin-petersohn approved these changes Jun 20, 2018

View reviewed changes

devin-petersohn merged commit 6bf48f4 into ray-project:master Jun 20, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DataFrame] Handling zero length partitions #2210

[DataFrame] Handling zero length partitions #2210

kunalgosar commented Jun 8, 2018 •

edited

Loading

AmplabJenkins commented Jun 8, 2018

AmplabJenkins commented Jun 11, 2018

pschafhalter Jun 11, 2018

pschafhalter Jun 11, 2018

kunalgosar Jun 11, 2018

devin-petersohn Jun 18, 2018

AmplabJenkins commented Jun 11, 2018

kunalgosar commented Jun 12, 2018

AmplabJenkins commented Jun 12, 2018

kunalgosar commented Jun 12, 2018

AmplabJenkins commented Jun 12, 2018

kunalgosar commented Jun 12, 2018

AmplabJenkins commented Jun 12, 2018

kunalgosar commented Jun 13, 2018

devin-petersohn commented Jun 18, 2018

AmplabJenkins commented Jun 18, 2018

devin-petersohn left a comment

devin-petersohn Jun 18, 2018

devin-petersohn Jun 18, 2018

devin-petersohn Jun 18, 2018

devin-petersohn Jun 18, 2018

AmplabJenkins commented Jun 20, 2018

kunalgosar commented Jun 20, 2018

devin-petersohn commented Jun 20, 2018

AmplabJenkins commented Jun 20, 2018

[DataFrame] Handling zero length partitions #2210

[DataFrame] Handling zero length partitions #2210

Conversation

kunalgosar commented Jun 8, 2018 • edited Loading

AmplabJenkins commented Jun 8, 2018

AmplabJenkins commented Jun 11, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jun 11, 2018

kunalgosar commented Jun 12, 2018

AmplabJenkins commented Jun 12, 2018

kunalgosar commented Jun 12, 2018

AmplabJenkins commented Jun 12, 2018

kunalgosar commented Jun 12, 2018

AmplabJenkins commented Jun 12, 2018

kunalgosar commented Jun 13, 2018

devin-petersohn commented Jun 18, 2018

AmplabJenkins commented Jun 18, 2018

devin-petersohn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AmplabJenkins commented Jun 20, 2018

kunalgosar commented Jun 20, 2018

devin-petersohn commented Jun 20, 2018

AmplabJenkins commented Jun 20, 2018

kunalgosar commented Jun 8, 2018 •

edited

Loading