-
Notifications
You must be signed in to change notification settings - Fork 155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Warn & replace dataframes with non-unique indexes #691
Merged
rapids-bot
merged 63 commits into
nv-morpheus:branch-23.03
from
dagardner-nv:david-warn-non-unique-686
Mar 17, 2023
Merged
Warn & replace dataframes with non-unique indexes #691
rapids-bot
merged 63 commits into
nv-morpheus:branch-23.03
from
dagardner-nv:david-warn-non-unique-686
Mar 17, 2023
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mdemoret-nv
requested changes
Feb 13, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change the destructor of MutableTableInfo
to LOG(ERROR)
instead of LOG(FATAL)
.
96a2ee0
to
d4b8761
Compare
dagardner-nv
commented
Mar 16, 2023
After many additions to this PR, here is the final list of changes. Breaking Changes
Additional Changes
|
mdemoret-nv
approved these changes
Mar 17, 2023
/merge |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
has_unique_index
&replace_non_unique_index
helper methods toMessageMeta
file_type
argument ofread_file_to_df
now has a default value ofAuto
DeserializeStage
checks for non-unique indexes and replaces them if needed.This comes at a performance cost in that the
DeserializeStage
needs to acquire the GIL in order to check if the Dataframe has a unique index, impacting users who never run into this issue. We could work around this by providing ano-check
argument to the stage, or we could do the check in the constructor ofMessageMeta
when we already have the GIL and can perform the check quite cheapely.Fixes #689
Fixes #686
Fixes #687
Fixes #286
Fixes #626
Fixes #393