Skip to content

Allow moving_function to access percentile sketches #49452

Closed
@polyfractal

Description

@polyfractal

If the moving_function pipeline agg could access and merge the internal percentiles sketch (HDRHisto or TDigest), it would allow users to construct moving windows of percentiles. E.g. bucket the results at 1 minute intervals and apply a percentiles agg per-bucket, then using a moving_function over 5 buckets and inside the script merge the sketches to produce a single sketch for that window of time.

When plotted, this gives a percentiles output of the last five minutes, every minute.

This is similar to what the cumulative cardinality agg does, except merging windows of percentiles together instead of cumulatively merging a cardinality sketch.

I think the main difficulty will be exposing this functionality in a safe/simple manner to painless. I think we would first need to allow percentiles objects as a valid input (instead of just numerics/multi-valued stats) and then build a dedicated painless function similar to the existing moving averages (e.g. MovingFunctions.percentiles(values))

If we can't find a good way to do this, we could always create a specialized "moving percentiles" pipeline agg but it would probably be better to extend moving_function first if possible.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions