Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-32723: [C++][Parquet] Add option to use LARGE* variants of binary types #35825

Open
wants to merge 69 commits into
base: main
Choose a base branch
from
Open

GH-32723: [C++][Parquet] Add option to use LARGE* variants of binary types #35825

wants to merge 69 commits into from

Commits on May 26, 2023

  1. able to read the file

    @arthurpassos
    arthurpassos committed May 26, 2023
    Configuration menu
    Copy the full SHA
    e5e96ec View commit details
    Browse the repository at this point in the history

  2. remove diff out

    @arthurpassos
    arthurpassos committed May 26, 2023
    Configuration menu
    Copy the full SHA
    b9b48f8 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2023

  1. intermediate stage, not working properly anymore..

    @arthurpassos
    arthurpassos committed May 29, 2023
    Configuration menu
    Copy the full SHA
    ae62954 View commit details
    Browse the repository at this point in the history

  2. still not working

    @arthurpassos
    arthurpassos committed May 29, 2023
    Configuration menu
    Copy the full SHA
    34917d5 View commit details
    Browse the repository at this point in the history

  3. able to read the file again

    @arthurpassos
    arthurpassos committed May 29, 2023
    Configuration menu
    Copy the full SHA
    835b07d View commit details
    Browse the repository at this point in the history

Commits on May 30, 2023

  1. move use_binary_large_variants to arrowreaderproperties

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    50427c6 View commit details
    Browse the repository at this point in the history

  2. cleanup a bit

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    df65ce7 View commit details
    Browse the repository at this point in the history

  3. back fromByteArray string & binary with setting

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    e826b8e View commit details
    Browse the repository at this point in the history

  4. some more adjustments

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    764ef98 View commit details
    Browse the repository at this point in the history

  5. revert some stuff

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    c6244ea View commit details
    Browse the repository at this point in the history

  6. revert some stuff

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    5a4bbb0 View commit details
    Browse the repository at this point in the history

  7. improvement

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    2d84e57 View commit details
    Browse the repository at this point in the history

  8. remove dictionary64

    @arthurpassos
    arthurpassos committed May 30, 2023
    Configuration menu
    Copy the full SHA
    90f14df View commit details
    Browse the repository at this point in the history

Commits on May 31, 2023

  1. use 64bit on largebytearray class and initialize binary_large_variant… 

    … bool to false
    @arthurpassos
    arthurpassos committed May 31, 2023
    Configuration menu
    Copy the full SHA
    b88b024 View commit details
    Browse the repository at this point in the history

  2. add chunked string map test

    @arthurpassos
    arthurpassos committed May 31, 2023
    Configuration menu
    Copy the full SHA
    0b53b05 View commit details
    Browse the repository at this point in the history

  3. add boolean comment

    @arthurpassos
    arthurpassos committed May 31, 2023
    Configuration menu
    Copy the full SHA
    f574e2e View commit details
    Browse the repository at this point in the history

  4. Make ChunkedRecordReader generic by using templates

    @arthurpassos
    arthurpassos committed May 31, 2023
    Configuration menu
    Copy the full SHA
    295e062 View commit details
    Browse the repository at this point in the history

  5. Make ByteArrayDictionaryReader generic with the use of templates

    @arthurpassos
    arthurpassos committed May 31, 2023
    Configuration menu
    Copy the full SHA
    25d7815 View commit details
    Browse the repository at this point in the history

  6. make arrowbinaryhelper generic

    @arthurpassos
    arthurpassos committed May 31, 2023
    Configuration menu
    Copy the full SHA
    fe8d67b View commit details
    Browse the repository at this point in the history

  7. Make PlainByteArrayDecoder generic

    @arthurpassos
    arthurpassos committed May 31, 2023
    Configuration menu
    Copy the full SHA
    35e5835 View commit details
    Browse the repository at this point in the history

Commits on Jun 1, 2023

  1. remove use_binary_large_variant from parquet reader properties

    @arthurpassos
    arthurpassos committed Jun 1, 2023
    Configuration menu
    Copy the full SHA
    9aff2f3 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. removed parquet::type::large_Byte_array

    @arthurpassos
    arthurpassos committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    eb850c4 View commit details
    Browse the repository at this point in the history

  2. small adjustment

    @arthurpassos
    arthurpassos committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    c2aab63 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2023

  1. remove largebytearray class

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    837ed6c View commit details
    Browse the repository at this point in the history

  2. simplify largebytearraytype a bit

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    35cdb99 View commit details
    Browse the repository at this point in the history

  3. simplify dictbytearraydecoderimpl a bit

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    a5000e1 View commit details
    Browse the repository at this point in the history

  4. remove one default argument

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    eb71c17 View commit details
    Browse the repository at this point in the history

  5. remove junk code

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    686a3f7 View commit details
    Browse the repository at this point in the history

  6. move use_binary_large_variant check inside frombytearray

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    a61fc32 View commit details
    Browse the repository at this point in the history

  7. simplify chunkedrecordreader a bit

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    e2600d0 View commit details
    Browse the repository at this point in the history

  8. simplify DictionaryRecordReaderImpl and fix DebugPrintState

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    3b86e23 View commit details
    Browse the repository at this point in the history

  9. simplify PlainByteArrayDecoderBase

    @arthurpassos
    arthurpassos committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    cc027b7 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2023

  1. remove some todos

    @arthurpassos
    arthurpassos committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    177db7a View commit details
    Browse the repository at this point in the history

  2. Add comment explaining why struct LargeByteArrayType instead of alias

    @arthurpassos
    arthurpassos committed Jun 7, 2023
    Configuration menu
    Copy the full SHA
    66223ee View commit details
    Browse the repository at this point in the history

Commits on Jun 8, 2023

  1. address some pr comments

    @arthurpassos
    arthurpassos committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    5cd39d8 View commit details
    Browse the repository at this point in the history

  2. address a few more comments

    @arthurpassos
    arthurpassos committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    1089010 View commit details
    Browse the repository at this point in the history

  3. remove arrow-type include & move binarylimit trait

    @arthurpassos
    arthurpassos committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    a6c42ee View commit details
    Browse the repository at this point in the history

  4. consolidate setdict

    @arthurpassos
    arthurpassos committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    15be2a2 View commit details
    Browse the repository at this point in the history

  5. apply clangformat

    @arthurpassos
    arthurpassos committed Jun 8, 2023
    Configuration menu
    Copy the full SHA
    8d5ba3d View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2023

  1. removed todos

    @arthurpassos
    arthurpassos committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    fd8f979 View commit details
    Browse the repository at this point in the history

  2. a bit more renaming

    @arthurpassos
    arthurpassos committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    a5736d5 View commit details
    Browse the repository at this point in the history

  3. address one mor comment

    @arthurpassos
    arthurpassos committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    b4ecd0d View commit details
    Browse the repository at this point in the history

  4. add overflow check in dict

    @arthurpassos
    arthurpassos committed Jun 9, 2023
    Configuration menu
    Copy the full SHA
    9e9dff9 View commit details
    Browse the repository at this point in the history

Commits on Jun 12, 2023

  1. address a few comments

    @arthurpassos
    arthurpassos committed Jun 12, 2023
    Configuration menu
    Copy the full SHA
    ae1db20 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2023

  1. use int32_t explicitly

    @arthurpassos
    arthurpassos committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    09a9eaf View commit details
    Browse the repository at this point in the history

  2. use template directly

    @arthurpassos
    arthurpassos committed Jun 14, 2023
    Configuration menu
    Copy the full SHA
    1664983 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2023

  1. use offset_type

    @arthurpassos
    arthurpassos committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    322319e View commit details
    Browse the repository at this point in the history

  2. address comments

    @arthurpassos
    arthurpassos committed Jun 15, 2023
    Configuration menu
    Copy the full SHA
    1775a7a View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2023

  1. address a few minor comments

    @arthurpassos
    arthurpassos committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    7f6e2bf View commit details
    Browse the repository at this point in the history

  2. fix DictDecoderImpl

    @arthurpassos
    arthurpassos committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    75fb615 View commit details
    Browse the repository at this point in the history

  3. add non overflow test

    @arthurpassos
    arthurpassos committed Jun 16, 2023
    Configuration menu
    Copy the full SHA
    0801267 View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2023

  1. string test

    @arthurpassos
    arthurpassos committed Jun 19, 2023
    Configuration menu
    Copy the full SHA
    7f09a16 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. address minor comments

    @arthurpassos
    arthurpassos committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    a8d20a4 View commit details
    Browse the repository at this point in the history

Commits on Jun 21, 2023

  1. use raw filereaderbuilder instead of adding a new openfile function

    @arthurpassos
    arthurpassos committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    5fcf4e1 View commit details
    Browse the repository at this point in the history

  2. rename test

    @arthurpassos
    arthurpassos committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    8901cbc View commit details
    Browse the repository at this point in the history

  3. update test file name

    @arthurpassos
    arthurpassos committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    dff017a View commit details
    Browse the repository at this point in the history

  4. update submodule?

    @arthurpassos
    arthurpassos committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    232e01f View commit details
    Browse the repository at this point in the history

  5. aply clang-format

    @arthurpassos
    arthurpassos committed Jun 21, 2023
    Configuration menu
    Copy the full SHA
    d7d76c6 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2023

  1. address minor comments

    @arthurpassos
    arthurpassos committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    90ceb07 View commit details
    Browse the repository at this point in the history

  2. delta & delta length for large*

    @arthurpassos
    arthurpassos committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    0394963 View commit details
    Browse the repository at this point in the history

  3. fix wrong if statements

    @arthurpassos
    arthurpassos committed Jun 22, 2023
    Configuration menu
    Copy the full SHA
    a8df2e7 View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2023

  1. Template member variable as well

    @arthurpassos
    arthurpassos committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    2bb3b14 View commit details
    Browse the repository at this point in the history

  2. add docstring

    @arthurpassos
    arthurpassos committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    c114d44 View commit details
    Browse the repository at this point in the history

  3. add LargeStringDictionary32Builder

    @arthurpassos
    arthurpassos committed Jun 23, 2023
    Configuration menu
    Copy the full SHA
    d1d5798 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2023

  1. address a few comments

    @arthurpassos
    arthurpassos committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    0eaa60f View commit details
    Browse the repository at this point in the history

  2. clang format

    @arthurpassos
    arthurpassos committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    1e642fa View commit details
    Browse the repository at this point in the history

  3. add binarypacked test for largebinaryvariant

    @arthurpassos
    arthurpassos committed Jun 26, 2023
    Configuration menu
    Copy the full SHA
    b299497 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2023

  1. Revert "add binarypacked test for largebinaryvariant" 

    This reverts commit b299497.
    @arthurpassos
    arthurpassos committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    2c23dd7 View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. only run largebinary tests if system is 64bit

    @arthurpassos
    arthurpassos committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    eca9d6f View commit details
    Browse the repository at this point in the history