Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ARROW] Arrow serialization should not introduce extra shuffle for outermost limit #4662

Closed
wants to merge 28 commits into from
Closed

[ARROW] Arrow serialization should not introduce extra shuffle for outermost limit #4662

wants to merge 28 commits into from

Commits on Apr 4, 2023

  1. arrow take

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    a584943 View commit details
    Browse the repository at this point in the history

  2. driver slice last batch

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    8593d85 View commit details
    Browse the repository at this point in the history

  3. refine

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    0088671 View commit details
    Browse the repository at this point in the history

  4. refactor

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    ed8c692 View commit details
    Browse the repository at this point in the history

  5. refactor and add ut

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    4212a89 View commit details
    Browse the repository at this point in the history

  6. add ut

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    6c5b1eb View commit details
    Browse the repository at this point in the history

  7. revert unnecessarily changes

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    ee5a756 View commit details
    Browse the repository at this point in the history

  8. unnecessarily changes

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    4e7ca54 View commit details
    Browse the repository at this point in the history

  9. infer row size by schema.defaultSize

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    885cf2c View commit details
    Browse the repository at this point in the history

  10. add docs

    @cfmcgrady
    cfmcgrady committed Apr 4, 2023
    Configuration menu
    Copy the full SHA
    25e4f05 View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. address comment

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    03d0747 View commit details
    Browse the repository at this point in the history

  2. reflective calla AdaptiveSparkPlanExec.finalPhysicalPlan

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    2286afc View commit details
    Browse the repository at this point in the history

  3. address comment

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    81886f0 View commit details
    Browse the repository at this point in the history

  4. refactor

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    e3bf84c View commit details
    Browse the repository at this point in the history

  5. SparkPlan.session -> SparkSession.active to adapt Spark-3.1.x

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    d70aee3 View commit details
    Browse the repository at this point in the history

  6. SparkArrowbasedOperationSuite adapt Spark-3.1.x

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    4cef204 View commit details
    Browse the repository at this point in the history

  7. fix

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    573a262 View commit details
    Browse the repository at this point in the history

  8. SparkArrowbasedOperationSuite adapt Spark-3.1.x

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    c83cf3f View commit details
    Browse the repository at this point in the history

  9. make toBatchIterator private

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    9ffb44f View commit details
    Browse the repository at this point in the history

  10. add offset support to adapt Spark-3.4.x

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    b72bc6f View commit details
    Browse the repository at this point in the history

  11. add ut

    @cfmcgrady
    cfmcgrady committed Apr 6, 2023
    Configuration menu
    Copy the full SHA
    22cc70f View commit details
    Browse the repository at this point in the history

Commits on Apr 7, 2023

  1. add isStaticConfigKey to adapt Spark-3.1.x

    @cfmcgrady
    cfmcgrady committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    8280783 View commit details
    Browse the repository at this point in the history

  2. address comment

    @cfmcgrady
    cfmcgrady committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    6d596fc View commit details
    Browse the repository at this point in the history

  3. limit = 0 test case

    @cfmcgrady
    cfmcgrady committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    6064ab9 View commit details
    Browse the repository at this point in the history

  4. SparkArrowbasedOperationSuite adapt Spark-3.1.x

    @cfmcgrady
    cfmcgrady committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    3700839 View commit details
    Browse the repository at this point in the history

  5. exclude rule OptimizeLimitZero

    @cfmcgrady
    cfmcgrady committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    facc13f View commit details
    Browse the repository at this point in the history

  6. finally close

    @cfmcgrady
    cfmcgrady committed Apr 7, 2023
    Configuration menu
    Copy the full SHA
    130bcb1 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2023

  1. close vector

    @cfmcgrady
    cfmcgrady committed Apr 8, 2023
    Configuration menu
    Copy the full SHA
    82c912e View commit details
    Browse the repository at this point in the history