Skip to content

[BUG] Min/Max on derived fields cannot be pushed down #4571

@qianheng-aws

Description

@qianheng-aws

Query Information

PPL Command/Query:

POST localhost:9200/_plugins/_ppl/_explain

source=opensearch_dashboards_sample_data_logs 
| eval RamMB=CEIL(machine.ram/1000000.0) 
| stats MIN(RamMB) as Min_RAM_MB

Expected Result:

"physical": "CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch_dashboards_sample_data_logs]], PushDownContext=[[AGGREGATION->rel#80:LogicalAggregate.NONE.[](input=RelSubset#79,group={},Min_RAM_MB=MIN($0)), LIMIT->10000]...

Actual Result:

"physical": "
EnumerableLimit
    EnumerableAggregate
        CalciteEnumerableIndexScan(table=[[OpenSearch, opensearch_dashboards_sample_data_logs]], PushDownContext=[[AGGREGATION->rel#80:LogicalAggregate.NONE.[](input=RelSubset#79,group={},Min_RAM_MB=MIN($0)), LIMIT->10000]...

Dataset Information

Dataset/Schema Type

  • OpenTelemetry (OTEL)
  • Simple Schema for Observability (SS4O)
  • Open Cybersecurity Schema Framework (OCSF)
  • Custom (details below)

Index Mapping

{
  "mappings": {
    "properties": {
      "machine": {
        "properties": {
          "ram": { "type": "long" },
          "os": { "type": "text" }
        }
      }
    }
  }
}

Sample Data

{
  "machine": {
    "ram": 8589934592,
    "os": "win 8"
  }
}

Bug Description

Issue Summary:
MIN/MAX on derived fields should be pushed down to be aggregate script, while failing to be pushed down with the latest code.

Steps to Reproduce:

  1. Create index with mapping like above
  2. Run explain command on the above PPL

Impact:

Environment Information

OpenSearch Version:
3.3.0-snapshot

Additional Details:
Seems to be introduced by this PR: #4281. Firstly found when investigating this issue: #4460

Screenshots

Metadata

Metadata

Assignees

No one assigned

    Labels

    PPLPiped processing languagebugSomething isn't workingpushdownpushdown related issues

    Type

    No type

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions