Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ARROW-201: [C++] Initial ParquetWriter implementation #78

Closed
wants to merge 10 commits into from

Conversation

xhochy
Copy link
Member

@xhochy xhochy commented May 16, 2016

No description provided.

@xhochy
Copy link
Member Author

xhochy commented May 16, 2016

Also with this PR comes read support for max_definition_level = 1.

NodePtr node_ =
GroupNode::Make("schema", Repetition::REQUIRED, std::vector<NodePtr>({pnode}));
return std::static_pointer_cast<GroupNode>(node_);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In testing more data types, we may wish to use templates and parameterized test cases to enable more code reuse

@wesm
Copy link
Member

wesm commented May 17, 2016

Finished reviewing -- looking good. Nice to get these first end-to-end tests working, so that we can then add the rest of the data types (and other missing features, e.g. compression) and work on read/write performance

@xhochy
Copy link
Member Author

xhochy commented May 17, 2016

Incorporated all comments, travis builds seem to be breaking in an unrelated manner.

What's missing is the initial check if we really have a flat column. Therefore I would need its definition.

@xhochy
Copy link
Member Author

xhochy commented May 17, 2016

Rebased and now also Travis is happy.

@wesm
Copy link
Member

wesm commented May 17, 2016

A column is flat if:

  • max repetition level is 0
  • for REQUIRED field, max def level is 0, for OPTIONAL the max def level is 1

does that seem right?

might be worth having a helper function for this

@xhochy
Copy link
Member Author

xhochy commented May 18, 2016

We now also have a check for a flat column.

I also wanted to add some DCHECKs (not needed for a merge but I like have code with assertions) but I get the following error if I added it somewhere in src/arrow/parquet/reader.cc:

error: ‘arrow::parquet::internal’ has not been declared

@wesm
Copy link
Member

wesm commented May 18, 2016

looks good, +1. let's look into the namespace issue in arrow/util/logging.h separately

@asfgit asfgit closed this in e0fb369 May 18, 2016
@xhochy xhochy deleted the arrow-201 branch March 7, 2017 16:16
praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
Fixed some bugs in the filter code path.
praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
Fixed some bugs in the filter code path.
praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
Fixed some bugs in the filter code path.
praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Aug 30, 2018
Fixed some bugs in the filter code path.
praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Sep 4, 2018
Fixed some bugs in the filter code path.
praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Sep 10, 2018
Fixed some bugs in the filter code path.
praveenbingo added a commit to praveenbingo/arrow that referenced this pull request Sep 10, 2018
Fixed some bugs in the filter code path.
xuechendi pushed a commit to xuechendi/arrow that referenced this pull request Aug 19, 2020
FelixYBW pushed a commit to FelixYBW/arrow that referenced this pull request Jan 26, 2022
* Initial commit

* Add int64 seed support

* Add unit test cases
zhztheplayer pushed a commit to zhztheplayer/arrow-1 that referenced this pull request Feb 8, 2022
* Initial commit

* Add int64 seed support

* Add unit test cases
zhztheplayer pushed a commit to zhztheplayer/arrow-1 that referenced this pull request Mar 3, 2022
* Initial commit

* Add int64 seed support

* Add unit test cases
rui-mo pushed a commit to rui-mo/arrow-1 that referenced this pull request Mar 23, 2022
* Initial commit

* Add int64 seed support

* Add unit test cases
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants