Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-4466: [Rust] [DataFusion] Add support for Parquet data source #3851

Closed
wants to merge 38 commits into from
Closed

ARROW-4466: [Rust] [DataFusion] Add support for Parquet data source #3851

wants to merge 38 commits into from

Commits on Mar 14, 2019

  1. Parquet datasource

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    10710a2 View commit details
    Browse the repository at this point in the history

  2. test

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    ff3e5b7 View commit details
    Browse the repository at this point in the history

  3. first parquet test passes

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    3a412b1 View commit details
    Browse the repository at this point in the history

  4. add test for reading strings from parquet

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    322fc87 View commit details
    Browse the repository at this point in the history

  5. save

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    eaddafb View commit details
    Browse the repository at this point in the history

  6. save

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    f46e6f7 View commit details
    Browse the repository at this point in the history

  7. convert to use row iter

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    aea9f8a View commit details
    Browse the repository at this point in the history

  8. add integration test

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    c3f71d7 View commit details
    Browse the repository at this point in the history

  9. revert to columnar reads

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    5ce3086 View commit details
    Browse the repository at this point in the history

  10. implement more parquet column types and tests

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    b4981ed View commit details
    Browse the repository at this point in the history

  11. add support for all primitive parquet types

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    6c3b7e2 View commit details
    Browse the repository at this point in the history

  12. code cleanup

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    debb2fb View commit details
    Browse the repository at this point in the history

  13. Remove invalid TODO comment

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    157512e View commit details
    Browse the repository at this point in the history

  14. update to use partition-aware changes from master

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    dddb7d7 View commit details
    Browse the repository at this point in the history

  15. remove println and unwrap

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    7e1a98f View commit details
    Browse the repository at this point in the history

  16. projection takes slice instead of vec

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    c56510e View commit details
    Browse the repository at this point in the history

  17. use parquet::reader::schema::parquet_to_arrow_schema

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    6457c36 View commit details
    Browse the repository at this point in the history

  18. revert temp debug change to error messages

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    e8aa784 View commit details
    Browse the repository at this point in the history

  19. return result if there are null values

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    607a29f View commit details
    Browse the repository at this point in the history

  20. replace read_column! macro with generic

    @nevi-me @andygrove
    nevi-me authored and andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    e6cbbaa View commit details
    Browse the repository at this point in the history

  21. immediately allocate vec

    @nevi-me @andygrove
    nevi-me authored and andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    3c711a5 View commit details
    Browse the repository at this point in the history

  22. fmt

    @nevi-me @andygrove
    nevi-me authored and andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    306d07a View commit details
    Browse the repository at this point in the history

  23. Remove unnecessary slice, fix null handling

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    5a3368c View commit details
    Browse the repository at this point in the history

  24. add date support

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    80cf303 View commit details
    Browse the repository at this point in the history

  25. handle nulls for binary data

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    1503855 View commit details
    Browse the repository at this point in the history

  26. null handling for int96

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    639e13e View commit details
    Browse the repository at this point in the history

  27. code cleanup

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    9d3047a View commit details
    Browse the repository at this point in the history

  28. remove println from tests

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    2aeea24 View commit details
    Browse the repository at this point in the history

  29. fix int96 conversion to read timestamps correctly

    @nevi-me
    nevi-me committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    02b2ed3 View commit details
    Browse the repository at this point in the history

  30. Merge pull request #2 from nevi-me/ARROW-4466 

    fix int96 conversion to read timestamps correctly
    @andygrove
    andygrove authored Mar 14, 2019
    Configuration menu
    Copy the full SHA
    023dc25 View commit details
    Browse the repository at this point in the history

  31. Clean up imports

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    1ec815b View commit details
    Browse the repository at this point in the history

  32. clean up handling of INT96 and DATE/TIME/TIMESTAMP types in schema co… 

    …nverter
    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    9b1308f View commit details
    Browse the repository at this point in the history

  33. Make INT32/64/96 handling consistent with C++ implementation

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    25d34ac View commit details
    Browse the repository at this point in the history

  34. Remove println from test

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    73aa934 View commit details
    Browse the repository at this point in the history

  35. fix timestamp nano issue

    @andygrove
    andygrove committed Mar 14, 2019
    Configuration menu
    Copy the full SHA
    204db83 View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2019

  1. move schema projection function from arrow into datafusion

    @andygrove
    andygrove committed Mar 15, 2019
    Configuration menu
    Copy the full SHA
    8d2df06 View commit details
    Browse the repository at this point in the history

  2. Remove hard-coded batch size, fix nits

    @andygrove
    andygrove committed Mar 15, 2019
    Configuration menu
    Copy the full SHA
    549c829 View commit details
    Browse the repository at this point in the history

  3. add test for reading small batches

    @andygrove
    andygrove committed Mar 15, 2019
    Configuration menu
    Copy the full SHA
    3158529 View commit details
    Browse the repository at this point in the history