Add "none" gap_policy to pipeline aggs #44516
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BucketScript was using the old-style parser and could easily be converted over to the newer static parser. Also adds a test for GapPolicy enum serialization
This adjusts the `buckets_path` parser so that pipeline aggs can select specific buckets (via their bucket keys) instead of fetching the entire set of buckets. This is useful for bucket_script in particular, which might want specific buckets for calculations. It's possible to workaround this with `filter` aggs, but the workaround is hacky and probably less performant. - Adjusts documentation - Adds a barebones AggregatorTestCase for bucket_script - Tweaks AggTestCase to use getMockScriptService() for reductions and pipelines. Previously pipelines could just pass in a script service for testing, but this didnt work for regular aggs. The new getMockScriptService() method fixes that issue, but needs to be used for pipelines too. This had a knock-on effect of touching MovFn, AvgBucket and ScriptedMetric
This adds a new gap policy, which does nothing. Gaps are presented as-is to the aggregator and up to the agg to decide what to do. Some aggs cannot handle gaps (derivatives, etc) and will treat the empty bucket as if it were missing. Other aggs (bucket_script, moving fn) will pass the empty bucket to the user for evaluation. This also adds a `params.doc_count` property which is accessible from pipeline scripts, allowing the user to differentiate between null values and empty buckets.
polyfractal
added
>enhancement
WIP
:Analytics/Aggregations
|
Pinging @elastic/es-analytics-geo |
|
Any news? Can't wait to test this and see if my exception is fixed 🙏 |
rjernst
added
the
Team:Analytics
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WIPThis adds a
"none"gap policy to pipeline aggregations, which is effectively a policy which does nothing to gaps. Missing/empty buckets are passed to the aggregation without any transformation. Some aggs can't handle null/missing values (derivative, etc) and so they will not emit any values for the affected buckets. Other aggs (likebucket_script) are executed and given the bucket value (null, NaN, etc) for the user to decide what to do with it.This also adds a
doc_countwhich is accessible from pipeline scripts, so that the user can differentiate between null-from-missing and null-from-document.This depends on #44179 for some changes to the test framework. MovingFn is currently broken because the queue doesn't support null values, but this should be alleviated once #44360 merges.
Closes: #27377, #42281