Skip to content

[7.x] Optimistic concurrency control for updating ingest pipelines #79364

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Oct 18, 2021

Conversation

danhermann
Copy link
Contributor

This commit adds optional optimistic concurrency control to the API for updating ingest pipelines. This is achieved through the addition of an if_version query parameter similar to the if_seq_no and if_primary_term parameters used for optimistic concurrency control in Elasticsearch. When this parameter is specified, the pipeline update succeeds only if there is an existing pipeline with a version that matches the specified version.

PUT /_ingest/pipeline/my_pipeline?if_version=5
{
  "version": 9,
  "processors": [
    ...
  ]
}

In the example above, the update succeeds only if "my_pipeline" exists with a version of 5. Any version can be specified in the body of the update so long as it is not the existing version. If no version is specified in the body of the update, the version is automatically set to <currentVersion> + 1. Pipelines that do not have a version attribute with a numeric value cannot be updated with the if_version parameter because checking for a null or missing version is not a meaningful guard against concurrent updates.

Note that any pipeline updates that do not include an if_version parameter will be processed unconditionally. Note also that there are no optimizations possible for no-op pipeline updates when the if_version parameter is specified because such an update will always change, at the very least, the version attribute of the pipeline and will therefore never be a no-op update.

Fixes #77031

Backport of #78551

@danhermann danhermann added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP backport v7.16.0 labels Oct 18, 2021
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Oct 18, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@danhermann
Copy link
Contributor Author

@elasticmachine run elasticsearch-ci/part-1

@danhermann danhermann merged commit 8ebc800 into elastic:7.x Oct 18, 2021
@danhermann danhermann deleted the backport_7x_78551_pipeline_occ branch October 18, 2021 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team v7.16.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants