Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for pylibcudf.DataType serialization #17352

Merged
merged 10 commits into from
Nov 28, 2024
Merged

Add support for pylibcudf.DataType serialization #17352

merged 10 commits into from
Nov 28, 2024

Conversation

pentschev
Copy link
Member

Description

For the multi-GPU cudf-polars work serializing pylibcudf.DataType will be required. This change implements the __reduce__ method and add appropriate tests.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@github-actions github-actions bot added Python Affects Python cuDF API. pylibcudf Issues specific to the pylibcudf package labels Nov 18, 2024
@pentschev pentschev added feature request New feature or request 2 - In Progress Currently a work in progress non-breaking Non-breaking change cudf.polars Issues specific to cudf.polars pylibcudf Issues specific to the pylibcudf package and removed Python Affects Python cuDF API. pylibcudf Issues specific to the pylibcudf package labels Nov 18, 2024
@vyasr
Copy link
Contributor

vyasr commented Nov 18, 2024

Could we just use __getstate__/__setstate__ instead? That is typically recommended since it is a simpler interface to use.

@pentschev
Copy link
Member Author

Could we just use __getstate__/__setstate__ instead? That is typically recommended since it is a simpler interface to use.

@vyasr according to this SO post we'll actually need to implement __getnewargs_ex__ as well due to __cinit__, which feels to me much more contrived than the current implementation, as we'll need to implement three new methods instead of only one. I don't mind doing that if you strongly prefer that solution, but I'd say to me that doesn't feel simpler at all and seems like it will increase maintenance burden rather than simplify it.

@pentschev @wence-
Make __reduce__ subclass-safe
3eeb98f 
Co-authored-by: Lawrence Mitchell <wence@gmx.li>
@github-actions github-actions bot added Python Affects Python cuDF API. and removed cudf.polars Issues specific to cudf.polars labels Nov 19, 2024
@pentschev pentschev mentioned this pull request Nov 19, 2024
Draft
3 tasks
@vyasr
Copy link
Contributor

vyasr commented Nov 19, 2024

I hadn't considered the additional complexity associated with it being an extension type with a cinit method. That's fine then, let's stick with reduce.

@vyasr vyasr changed the base branch from branch-24.12 to branch-25.02 November 20, 2024 16:48
@vyasr
Copy link
Contributor

vyasr commented Nov 20, 2024

Bumping to 25.02.

@pentschev pentschev added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Nov 25, 2024
@pentschev pentschev marked this pull request as ready for review November 25, 2024 22:31
@pentschev pentschev requested a review from a team as a code owner November 25, 2024 22:31
@pentschev pentschev requested review from bdice and Matt711 November 25, 2024 22:31
Matt711
Matt711 approved these changes Nov 25, 2024
View reviewed changes
Copy link
Contributor

@Matt711 Matt711 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, some tiny suggestions

python/pylibcudf/pylibcudf/tests/test_serialize.py Outdated Show resolved Hide resolved
python/pylibcudf/pylibcudf/tests/test_serialize.py Outdated

@pytest.mark.parametrize("tid", [t for t in type_id])
def test_reduce(tid):
dt = DataType(tid, 0)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
dt = DataType(tid, 0)
dt = DataType(tid)

python/pylibcudf/pylibcudf/tests/test_serialize.py Outdated Show resolved Hide resolved
python/pylibcudf/pylibcudf/tests/test_serialize.py Outdated Show resolved Hide resolved
wence-
wence- requested changes Nov 26, 2024
View reviewed changes
Copy link
Contributor

@wence- wence- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Serialization looks fine, but I think we can simplify the tests.

python/pylibcudf/pylibcudf/tests/test_serialize.py Outdated Show resolved Hide resolved
python/pylibcudf/pylibcudf/tests/test_serialize.py Outdated Show resolved Hide resolved
@pentschev
Copy link
Member Author

Thanks @wence- and @Matt711 for the reviews. Those have been addressed now mostly with Lawrence's suggestions that simplified the tests substantially. Would you mind having another look?

wence-
wence- approved these changes Nov 26, 2024
View reviewed changes
Copy link
Contributor

@wence- wence- left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Peter

@pentschev pentschev added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 3 - Ready for Review Ready for review by team labels Nov 26, 2024
Matt711
Matt711 approved these changes Nov 28, 2024
View reviewed changes
Copy link
Contributor

@Matt711 Matt711 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks

@pentschev
Copy link
Member Author

/merge

@rapids-bot rapids-bot bot merged commit 891a865 into rapidsai:branch-25.02 Nov 28, 2024
105 checks passed
@pentschev
Copy link
Member Author

Thanks all for reviews!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Matt711 Matt711 Matt711 approved these changes

@wence- wence- wence- approved these changes

@bdice bdice Awaiting requested review from bdice bdice is a code owner automatically assigned from rapidsai/cudf-python-codeowners

Assignees

@pentschev pentschev

Labels
5 - Ready to Merge Testing and reviews complete, ready to merge feature request New feature or request non-breaking Non-breaking change pylibcudf Issues specific to the pylibcudf package Python Affects Python cuDF API.
Projects
cuDF Python
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@pentschev @vyasr @wence- @Matt711