Skip to content

[FEATURE] PPL min/max enhancements for non-numeric values #4050

@dai-chen

Description

@dai-chen

Is your feature request related to a problem?

Currently max(<value>) and min(<value>) in PPL behave numerically and don’t match SPL’s behavior on non-numeric and mixed type fields.

What solution would you like?

  1. Align PPL min/max function with SPL's ordering rule in https://help.splunk.com/en/splunk-enterprise/search/spl-search-reference/10.0/statistical-and-charting-functions/aggregate-functions#usage-6.
  2. Pushdown optimization: DSL min/max aggregation doesn't support non-numeric fields. Probably we need to decide to pushdown to min/max or script aggregation based on input type [TBD].

What alternatives have you considered?
N/A

Do you have any additional context?

Here is the test result with PPL Calcite v3:

source=accounts | stats max(gender);
TransportError(500, 'SearchPhaseExecutionException', {'error': {'reason':
  'Error occurred in OpenSearch engine: all shards failed', 'details': 'Shard[0]: java.lang.IllegalArgumentException: 
    Field [gender.keyword] of type [keyword] is not supported for aggregation [max]\n\n
      For more details, please send request for Json format to see the raw response from OpenSearch engine.',
        'type': 'SearchPhaseExecutionException'}, 'status': 400})

Caused by: java.lang.IllegalArgumentException: Field [gender.keyword] of type [keyword] is not supported for aggregation [max]
        at org.opensearch.search.aggregations.support.ValuesSourceRegistry.getAggregator(ValuesSourceRegistry.java:202) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:417) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.aggregations.support.ValuesSourceAggregationBuilder$LeafOnly.doBuild(ValuesSourceAggregationBuilder.java:149) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:155) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:462) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.SearchService.parseSource(SearchService.java:1524) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.SearchService.createContext(SearchService.java:1207) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.SearchService.executeQueryPhase(SearchService.java:772) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:738) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:74) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.action.ActionRunnable$2.doRun(ActionRunnable.java:89) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.threadpool.TaskAwareRunnable.doRun(TaskAwareRunnable.java:78) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:59) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:975) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-3.2.0-SNAPSHOT.jar:3.2.0-SNAPSHOT]
        ... 3 more

Metadata

Metadata

Assignees

Labels

PPLPiped processing languageenhancementNew feature or requestv3.3.0

Type

No type

Projects

Status

New

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions