Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't try to infer nulls in CSV schema inference #2859

Closed
Dandandan opened this issue Oct 12, 2022 · 1 comment · Fixed by #2860
Closed

Don't try to infer nulls in CSV schema inference #2859

Dandandan opened this issue Oct 12, 2022 · 1 comment · Fixed by #2860
Labels
arrow Changes to the arrow crate bug

Comments

@Dandandan
Copy link
Contributor

Describe the bug

This came up in apache/datafusion#3783

I believe it is safer choice to infer nullable types always, and the benefit of having non-null inferred is relatively limited.

To Reproduce

Infer csv with nulls that occur after beyond the number of rows schema inference is using.

Expected behavior

Additional context
@Dandandan Dandandan added the bug label Oct 12, 2022
@Dandandan Dandandan mentioned this issue Oct 12, 2022
Merged
@alamb alamb added the arrow Changes to the arrow crate label Oct 14, 2022
@alamb
Copy link
Contributor

alamb commented Oct 14, 2022

label_issue.py automatically added labels {'arrow'} from #2860

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arrow Changes to the arrow crate bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Don't try to infer nullability in CSV reader Dandandan/arrow-rs
2 participants
@Dandandan @alamb