Skip to content

[BUG] Limit does not take effect in aggregation pushdown after sort #4296

@yuancu

Description

@yuancu

Query Information

PPL Command/Query:

source=opensearch-sql_test_index_bank | stats count() by age | sort -age | head 3

Expected Result: truncated to the top 3 rows

count() age
1 39
2 36
1 34

Actual Result: head 3 did not take effect

count() age
1 39
2 36
1 34
1 33
1 32
1 28

Dataset Information

Dataset/Schema Type

Sample Data

{
  "account_number": 1,
  "balance": 39225,
  "firstname": "Amber JOHnny",
  "lastname": "Duke Willmington",
  "age": 32,
  "gender": "M",
  "address": "880 Holmes Lane",
  "employer": "Pyrami",
  "email": "amberduke@pyrami.com",
  "city": "Brogan",
  "state": "IL",
  "male": true,
  "birthdate": "2017-10-23"
}

Bug Description

Issue Summary:

Steps to Reproduce:

  1. Adjust settings to enable Calcite and turn on pushdown
    PUT _cluster/settings
    {
       "persistent":{
          "plugins.calcite.enabled": true,
          "plugins.calcite.fallback.allowed": false
       }
    }
    
    PUT _cluster/settings
    {
       "persistent":{
          "plugins.calcite.pushdown.enabled": true
       }
    }
  2. Run the following query:
    POST /_plugins/_ppl
    {
      "query" : "source=opensearch-sql_test_index_bank | stats count() by age | sort -age | head 3"
    }

Impact:

Environment Information

OpenSearch Version: 3.2.0, latest commit: c518022

Additional Details:

  • Execution plan for the mentioned query:
EnumerableCalc(expr#0..1=[{inputs}], count()=[$t1], age=[$t0])
  CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch-sql_test_index_bank]], PushDownContext=[[AGGREGATION->rel#4341:LogicalAggregate.NONE.[](input=RelSubset#4340,group={0},count()=COUNT()), SORT->[0 DESC LAST], LIMIT->3, LIMIT->10000], OpenSearchRequestBuilder(sourceBuilder={"from":0,"size":0,"timeout":"1m","aggregations":{"composite_buckets":{"composite":{"size":1000,"sources":[{"age":{"terms":{"field":"age","missing_bucket":true,"missing_order":"last","order":"desc"}}}]},"aggregations":{"count()":{"value_count":{"field":"_index"}}}}}}, requestedTotalSize=2147483647, pageSize=null, startFrom=0)])

Metadata

Metadata

Assignees

Labels

PPLPiped processing languagebugSomething isn't workingcalcitecalcite migration releated

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions