Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

new ppl-spark logical plan translation #31

Closed
wants to merge 26 commits into from
Closed

new ppl-spark logical plan translation #31

wants to merge 26 commits into from

Commits on Aug 23, 2023

  1. adding support for containerized flint with spark / Livy docker-compo… 

    …se.yml

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    62c12cc View commit details
    Browse the repository at this point in the history

  2. adding support for containerized flint with spark / Livy docker-compo… 

    …se.yml

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Aug 23, 2023
    Configuration menu
    Copy the full SHA
    9e6ecfc View commit details
    Browse the repository at this point in the history

Commits on Sep 1, 2023

  1. adding support for containerized flint with spark / Livy docker-compo… 

    …se.yml

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    0808ea5 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into containerize_flint

    @YANG-DB
    YANG-DB committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    1b2ae06 View commit details
    Browse the repository at this point in the history

  3. adding support for containerized flint with spark / Livy docker-compo… 

    …se.yml

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    91defa0 View commit details
    Browse the repository at this point in the history

  4. update ppl ast builder 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    0febc09 View commit details
    Browse the repository at this point in the history

  5. add ppl ast components 

    add ppl statement logical plan elements
add ppl parser components
add ppl expressions components

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    18cd83f View commit details
    Browse the repository at this point in the history

  6. populate ppl test suit for covering different types of PPL queries 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    605f1bf View commit details
    Browse the repository at this point in the history

  7. update additional tests 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 1, 2023
    Configuration menu
    Copy the full SHA
    d54530d View commit details
    Browse the repository at this point in the history

Commits on Sep 6, 2023

  1. separate ppl-spark code into a dedicated module 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    72dc5f7 View commit details
    Browse the repository at this point in the history

  2. add ppl translation of simple filter and data-type literal expression 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    d953b19 View commit details
    Browse the repository at this point in the history

  3. remove none-used ppl ast builder 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    9fce31e View commit details
    Browse the repository at this point in the history

  4. add log-plan test results validation 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    a299bdf View commit details
    Browse the repository at this point in the history

  5. add support for multiple table selection using union 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 6, 2023
    Configuration menu
    Copy the full SHA
    019f690 View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2023

  1. add support for multiple table selection using union 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    0c7ccec View commit details
    Browse the repository at this point in the history

  2. update sbt with new IT test suite for PPL module 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    14fa7e5 View commit details
    Browse the repository at this point in the history

  3. update ppl IT suite test 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    d55b774 View commit details
    Browse the repository at this point in the history

  4. update ppl IT suite dependencies 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    8bbe0d9 View commit details
    Browse the repository at this point in the history

  5. add tests for ppl IT with 

     -  source = $testTable
 -  source = $testTable | fields name, age
 -  source = $testTable age=25 | fields name, age

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    af065f7 View commit details
    Browse the repository at this point in the history

  6. update literal transformations according to catalyst's convention 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    5819dc7 View commit details
    Browse the repository at this point in the history

  7. separate unit-tests into a dedicated file per each test category 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    7db7213 View commit details
    Browse the repository at this point in the history

  8. add IT tests for additional filters 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    32573ab View commit details
    Browse the repository at this point in the history

  9. mark unsatisfied tests as ignored until supporting code is ready 

    Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 7, 2023
    Configuration menu
    Copy the full SHA
    eec0e4a View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2023

  1. add README.md design and implementation details 

    add AggregateFunction translation & tests
remove unused DSL builder

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Sep 8, 2023
    Configuration menu
    Copy the full SHA
    3f9d9d1 View commit details
    Browse the repository at this point in the history

Commits on Oct 4, 2023

  1. Merge branch 'main' into containerize_flint 

    # Conflicts:
#	build.sbt
#	flint-spark-integration/src/test/scala/org/opensearch/flint/spark/skipping/FlintSparkSkippingIndexSuite.scala
#	ppl-spark-integration/README.md
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ast/expression/Span.java
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystPlanContext.java
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/CatalystQueryPlanVisitor.java
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/utils/AggregatorTranslator.java
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/utils/ComparatorTransformer.java
#	ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/utils/DataTypeTransformer.java
#	ppl-spark-integration/src/main/scala/org/opensearch/flint/spark/FlintPPLSparkExtensions.scala
#	ppl-spark-integration/src/main/scala/org/opensearch/flint/spark/ppl/FlintSparkPPLParser.scala
#	ppl-spark-integration/src/main/scala/org/opensearch/flint/spark/ppl/PPLSyntaxParser.scala
#	ppl-spark-integration/src/test/scala/org/opensearch/flint/spark/ppl/LogicalPlanTestUtils.scala
#	ppl-spark-integration/src/test/scala/org/opensearch/flint/spark/ppl/PPLLogicalPlanAggregationQueriesTranslatorTestSuite.scala
#	ppl-spark-integration/src/test/scala/org/opensearch/flint/spark/ppl/PPLLogicalPlanFiltersTranslatorTestSuite.scala
    @YANG-DB
    YANG-DB committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    2ae65f4 View commit details
    Browse the repository at this point in the history

  2. update docker-compose.yml and different docs relating to te local spa… 

    …rk-flint tests

Signed-off-by: YANGDB <yang.db.dev@gmail.com>
    @YANG-DB
    YANG-DB committed Oct 4, 2023
    Configuration menu
    Copy the full SHA
    6e3437d View commit details
    Browse the repository at this point in the history