Adds a new Rollup Action #64900

talevy · 2020-11-10T23:40:10Z

This commit adds a new endpoint for Rollup V2, a new way to rollup indices.

Instead of relying on a cron-job, this action will rollup a whole index on the spot.

When an index is rolled up using a Rollup Config, it does the following

check that original index is read-only
runs an aggregation and indexes results into a temporary hidden rollup index
"resizes" the temporary index into the final rollup-index (in-place segment pointer juggling)
adds RollupMetadata about the rollup-group (keyed by original index name) and adds custom index-metadata with data about what the rollup index's original index is so that its group information in RollupMetadata can be looked up

example usage:

POST /index/_rollup
{
    "rollup_index": "index_rolled",
    "groups": {
        "date_histogram": {
            "field": "date",
            "calendar_interval": "1M"
        },
        "terms": {
            "fields": ["unit"]
        }
    },
    "metrics": [
        {
            "field": "temperature",
            "metrics": ["sum"]
        }
    ]
}

elasticmachine · 2020-11-10T23:40:12Z

Pinging @elastic/es-analytics-geo (:Analytics/Rollup)

talevy · 2020-11-12T00:46:39Z

run elasticsearch-ci/2
run elasticsearch-ci/bwc

talevy · 2020-11-12T00:49:51Z

For the reviewers @csoulios and @not-napoleon, I am still working on making more of the indexer logic unit-testable. I would love some feedback on the general process and any large gotchyas you may see. There is a lot of state change and index dancing. I do intend to make these parts a bit more resilient, but that may not be a part of this PR pending further discussion! thanks!

This commit adds a new endpoint for Rollup V2, a new way to rollup indices. Instead of relying on a cron-job, this action will rollup a whole index on the spot. When an index is rolled up using a Rollup Config, it does the following 1. check that original index is read-only 2. runs an aggregation and indexes results into a temporary hidden rollup index 3. "resizes" the temporary index into the final rollup-index (in-place segment pointer juggling) 4. adds RollupMetadata about the rollup-group (keyed by original index name) and adds custom index-metadata with data about what the rollup index's original index is so that its group information in RollupMetadata can be looked up example usage: ``` POST /_rollup_vtwo/index { "rollup_index": "index_rolled", "groups": { "date_histogram": { "field": "date", "calendar_interval": "1M" }, "terms": { "fields": ["unit"] } }, "metrics": [ { "field": "temperature", "metrics": ["sum"] } ] } ```

not-napoleon

I left a few thoughts, but I don't think any of them are blockers to merging.

server/src/main/java/org/elasticsearch/cluster/ClusterModule.java

x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/v2/RollupV2Action.java

...k/plugin/rollup/src/main/java/org/elasticsearch/xpack/rollup/v2/TransportRollupV2Action.java

This commit adds a new endpoint for Rollup V2, a new way to rollup indices. Instead of relying on a cron-job, this action will rollup a whole index on the spot. When an index is rolled up using a Rollup Config, it does the following 1. check that original index is read-only 2. runs an aggregation and indexes results into a temporary hidden rollup index 3. "resizes" the temporary index into the final rollup-index (in-place segment pointer juggling) 4. adds RollupMetadata about the rollup-group (keyed by original index name) and adds custom index-metadata with data about what the rollup index's original index is so that its group information in RollupMetadata can be looked up example usage: ``` POST /_rollup_vtwo/index { "rollup_index": "index_rolled", "groups": { "date_histogram": { "field": "date", "calendar_interval": "1M" }, "terms": { "fields": ["unit"] } }, "metrics": [ { "field": "temperature", "metrics": ["sum"] } ] } ``` relates elastic#42720.

This commit adds a new endpoint for Rollup V2, a new way to rollup indices. Instead of relying on a cron-job, this action will rollup a whole index on the spot. When an index is rolled up using a Rollup Config, it does the following 1. check that original index is read-only 2. runs an aggregation and indexes results into a temporary hidden rollup index 3. "resizes" the temporary index into the final rollup-index (in-place segment pointer juggling) 4. adds RollupMetadata about the rollup-group (keyed by original index name) and adds custom index-metadata with data about what the rollup index's original index is so that its group information in RollupMetadata can be looked up example usage: ``` POST /_rollup_vtwo/index { "rollup_index": "index_rolled", "groups": { "date_histogram": { "field": "date", "calendar_interval": "1M" }, "terms": { "fields": ["unit"] } }, "metrics": [ { "field": "temperature", "metrics": ["sum"] } ] } ``` relates #42720.

This PR implements downsampling operation on time series indices. The PR creates a _rollup endpoint that allows users to downsample an index and can be accessed by the following call: POST /<source_index>/_rollup/<rollup_index> { "fixed_interval": "1d" } Requirements An index can be downsampled if all of the following requirements are met: Must be a time series index (have the index.mode: time_series index setting) Must not be writeable (have the index.blocks.write: true index setting) Must have dimension fields marked with mapping parameter time_series_dimension: true Must have metric fields marked with mapping parameter time_series_metric Relates to #74660 Fixes #65769 Fixes #69799 Finally, this PR is based on the code written for #64900

This PR implements downsampling operation on time series indices. The PR creates a _rollup endpoint that allows users to downsample an index and can be accessed by the following call: POST /<source_index>/_rollup/<rollup_index> { "fixed_interval": "1d" } Requirements An index can be downsampled if all of the following requirements are met: Must be a time series index (have the index.mode: time_series index setting) Must not be writeable (have the index.blocks.write: true index setting) Must have dimension fields marked with mapping parameter time_series_dimension: true Must have metric fields marked with mapping parameter time_series_metric Relates to elastic#74660 Fixes elastic#65769 Fixes elastic#69799 Finally, this PR is based on the code written for elastic#64900

talevy added >feature :StorageEngine/Rollup Turn fine-grained time-based data into coarser-grained data v8.0.0 Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels Nov 10, 2020

talevy mentioned this pull request Nov 10, 2020

Refactor rollups meta (AKA Rollup V2) #42720

Closed

21 tasks

talevy force-pushed the rollupv2action branch from 62b76a5 to be90dd3 Compare November 12, 2020 00:06

talevy requested review from csoulios and not-napoleon November 12, 2020 00:48

talevy added 2 commits November 18, 2020 16:01

fix config serialization tests

1a3e115

talevy force-pushed the rollupv2action branch from be90dd3 to 1a3e115 Compare November 19, 2020 00:01

not-napoleon approved these changes Nov 19, 2020

View reviewed changes

talevy added 2 commits November 19, 2020 11:40

Merge remote-tracking branch 'elastic/master' into rollupv2action

55a3087

fix merge issue where aggregate_metric was removed

49796c2

talevy changed the title ~~Adds a new Rollup V2 Action~~ Adds a new Rollup Action Nov 19, 2020

talevy added 6 commits November 19, 2020 15:48

rename endpoint from rollupv2 -> rollup

35d5dc1

fix checkstyle

8af508a

use valid rest path

96f4840

Merge remote-tracking branch 'elastic/master' into rollupv2action

32d5857

cleanup adding of new rollup groups

a395a03

fix checkstyle

219e356

talevy added >non-issue and removed >feature labels Nov 23, 2020

talevy merged commit 06da3d7 into elastic:master Nov 23, 2020

talevy deleted the rollupv2action branch November 23, 2020 21:04

talevy added v7.11.0 backport pending labels Nov 23, 2020

jrodewig mentioned this pull request Nov 24, 2020

[DOCS] Add rollup V2 API docs #65398

Merged

talevy removed the backport pending label Nov 24, 2020

talevy mentioned this pull request Dec 2, 2020

Rollup Action must throw exception when rollup index already exists #65769

Closed

jakelandis added v8.0.0-alpha1 and removed v8.0.0 labels Jul 26, 2021

csoulios mentioned this pull request Apr 14, 2022

TSDB: Implement downsampling on time-series indices #85708

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Adds a new Rollup Action #64900

Adds a new Rollup Action #64900

Uh oh!

talevy commented Nov 10, 2020 •

edited

Loading

Uh oh!

elasticmachine commented Nov 10, 2020

Uh oh!

talevy commented Nov 12, 2020

Uh oh!

talevy commented Nov 12, 2020 •

edited

Loading

Uh oh!

not-napoleon left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Adds a new Rollup Action #64900

Adds a new Rollup Action #64900

Uh oh!

Conversation

talevy commented Nov 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticmachine commented Nov 10, 2020

Uh oh!

talevy commented Nov 12, 2020

Uh oh!

talevy commented Nov 12, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

not-napoleon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

talevy commented Nov 10, 2020 •

edited

Loading

talevy commented Nov 12, 2020 •

edited

Loading