Skip to content

[DataFrame] Refactor __delitem__ #2080

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
May 18, 2018
Merged

[DataFrame] Refactor __delitem__ #2080

merged 2 commits into from
May 18, 2018

Conversation

simon-mo
Copy link
Contributor

What do these changes do?

Refactor __delitem__ for it to operate on block partitions only for efficiency. Also

  • Address an issue in index_metadata relating to arrow sealed object mutation
  • Add related tests.

Related issue number

Address issue #2027

simon-mo
simon-mo commented May 17, 2018
View reviewed changes
python/ray/dataframe/dataframe.py
@@ -4852,15 +4848,18 @@ def del_helper(df, to_delete):
# Cast cols as pd.Series as duplicate columns mean result may be
# np.int64 or pd.Series
col_parts_to_del = \
pd.Series(self._col_metadata[key, 'partition']).unique()
pd.Series(del_df['partition'].copy()).unique()
Copy link
Contributor Author

@simon-mo simon-mo May 17, 2018 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy is necessary because unique will mutate the underlying data (🤔). If the underlying data comes from arrow sealed object, a "Buffer is read-only" exception will be raised.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of the caching and memoizing that Pandas does mutates the objects. In the future, hopefully we can get rid of the need for the copy.

simon-mo
simon-mo commented May 17, 2018
View reviewed changes
python/ray/dataframe/index_metadata.py
@@ -203,15 +203,15 @@ def reset_partition_coords(self, partitions=None):
# updated as well.
try:
self._coord_df.loc[partition_mask,
'index_within_partition'] = [
p for p in range(sum(partition_mask))]
'index_within_partition'] = np.arange(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The previous method will prevent consecutive del calls:

del df['col1']
del df['col2'] # Fail because the col_metadata contains float

It seems setting with loc requires extra caution here. That's why I added type casting.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/5447/
Test PASSed.

devin-petersohn
devin-petersohn approved these changes May 17, 2018
View reviewed changes
Copy link
Member

@devin-petersohn devin-petersohn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great fix, thanks!

python/ray/dataframe/dataframe.py
@@ -4852,15 +4848,18 @@ def del_helper(df, to_delete):
# Cast cols as pd.Series as duplicate columns mean result may be
# np.int64 or pd.Series
col_parts_to_del = \
pd.Series(self._col_metadata[key, 'partition']).unique()
pd.Series(del_df['partition'].copy()).unique()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A lot of the caching and memoizing that Pandas does mutates the objects. In the future, hopefully we can get rid of the need for the copy.

@devin-petersohn devin-petersohn merged commit 0b07602 into ray-project:master May 18, 2018
@devin-petersohn
Copy link
Member

Merged, thanks @simon-mo!

@devin-petersohn devin-petersohn mentioned this pull request May 18, 2018
Closed
alok added a commit to alok/ray that referenced this pull request May 18, 2018
@alok
Merge branch 'master' into comps
0285567 
* master: (22 commits)
  [xray] Fix bug in updating actor execution dependencies (ray-project#2064)
  [DataFrame] Refactor __delitem__ (ray-project#2080)
  [xray] Better error messaging when pulling from self. (ray-project#2068)
  Use source code in hash where possible (fix ray-project#2089) (ray-project#2090)
  Functions for flushing done tasks and evicted objects. (ray-project#2033)
  Fix compilation error for RAY_USE_NEW_GCS with latest clang. (ray-project#2086)
  [xray] Corrects Error Handling During Push and Pull. (ray-project#2059)
  [xray] Sophisticated task dependency management (ray-project#2035)
  Support calling positional arguments by keyword (fix ray-project#998) (ray-project#2081)
  [DataFrame] Improve performance of iteration methods (ray-project#2026)
  [DataFrame] Implement to_csv (ray-project#2014)
  [xray] Lineage cache only requests notifications about remote parent tasks (ray-project#2066)
  [rllib] Add magic methods for rollouts (ray-project#2024)
  [DataFrame] Allows DataFrame constructor to take in another DataFrame (ray-project#2072)
  Pin Pandas version for Travis to 0.22 (ray-project#2075)
  Fix python linting (ray-project#2076)
  [xray] Fix GCS table prefixes (ray-project#2065)
  Some tests for _submit API. (ray-project#2062)
  [rllib] Queue lib for python 2.7 (ray-project#2057)
  [autoscaler] Remove faulty assert that breaks during downscaling, pull configs from env (ray-project#2006)
  ...
alok added a commit to alok/ray that referenced this pull request May 21, 2018
@alok
Merge branch 'master' into fix-a3c-2
662eaa5 
* master: (24 commits)
  Performance fix (ray-project#2110)
  Use flake8-comprehensions (ray-project#1976)
  Improve error message printing and suppression. (ray-project#2104)
  [rllib] [doc] Broken link in ddpg doc
  YAPF, take 3 (ray-project#2098)
  [rllib] rename async -> _async (ray-project#2097)
  fix unused lambda capture (ray-project#2102)
  [xray] Use pubsub instead of timeout for ObjectManager Pull. (ray-project#2079)
  [DataFrame] Update _inherit_docstrings (ray-project#2085)
  [JavaWorker] Changes to the build system for support java worker (ray-project#2092)
  [xray] Fix bug in updating actor execution dependencies (ray-project#2064)
  [DataFrame] Refactor __delitem__ (ray-project#2080)
  [xray] Better error messaging when pulling from self. (ray-project#2068)
  Use source code in hash where possible (fix ray-project#2089) (ray-project#2090)
  Functions for flushing done tasks and evicted objects. (ray-project#2033)
  Fix compilation error for RAY_USE_NEW_GCS with latest clang. (ray-project#2086)
  [xray] Corrects Error Handling During Push and Pull. (ray-project#2059)
  [xray] Sophisticated task dependency management (ray-project#2035)
  Support calling positional arguments by keyword (fix ray-project#998) (ray-project#2081)
  [DataFrame] Improve performance of iteration methods (ray-project#2026)
  ...
alok added a commit to alok/ray that referenced this pull request May 24, 2018
@alok
Merge branch 'fix-a3c-torch' into trpo
9aac5bd 
* fix-a3c-torch: (37 commits)
  Add missing channel major
  Use correct filter size
  Add TODO
  Fix shape errors
  fmt
  Performance fix (ray-project#2110)
  Use flake8-comprehensions (ray-project#1976)
  Improve error message printing and suppression. (ray-project#2104)
  [rllib] [doc] Broken link in ddpg doc
  YAPF, take 3 (ray-project#2098)
  [rllib] rename async -> _async (ray-project#2097)
  fix unused lambda capture (ray-project#2102)
  [xray] Use pubsub instead of timeout for ObjectManager Pull. (ray-project#2079)
  [DataFrame] Update _inherit_docstrings (ray-project#2085)
  [JavaWorker] Changes to the build system for support java worker (ray-project#2092)
  [xray] Fix bug in updating actor execution dependencies (ray-project#2064)
  [DataFrame] Refactor __delitem__ (ray-project#2080)
  [xray] Better error messaging when pulling from self. (ray-project#2068)
  Use source code in hash where possible (fix ray-project#2089) (ray-project#2090)
  Functions for flushing done tasks and evicted objects. (ray-project#2033)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@devin-petersohn devin-petersohn devin-petersohn approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@simon-mo @AmplabJenkins @devin-petersohn