Skip to content

Updating ingest pipeline without changes is no-op #78196

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 22, 2021

Conversation

danhermann
Copy link
Contributor

Fixes #77382.

@danhermann danhermann added >enhancement :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP v8.0.0 v7.16.0 labels Sep 22, 2021
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Sep 22, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

Copy link
Member

@martijnvg martijnvg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

}
ingestService.putPipeline(ingestInfos, request, listener);
}, listener::onFailure));
nodesInfoRequest.clear().addMetric(NodesInfoRequest.Metric.INGEST.metricName());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really part of your change, but since it is touched now, maybe avoid chaining? And instead just:

nodesInfoRequest.clear();
nodesInfoRequest..addMetric(NodesInfoRequest.Metric.INGEST.metricName());

I think that is easier to read.

if (currentIngestMetadata != null && currentIngestMetadata.getPipelines().containsKey(request.getId())) {
pipelineConfig = XContentHelper.convertToMap(request.getSource(), false, request.getXContentType()).v2();
var currentPipeline = currentIngestMetadata.getPipelines().get(request.getId());
if (currentPipeline.getConfigAsMap().equals(pipelineConfig)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was wondering whether Maps.deepEquals(...) be used here? But I don't think that is the case since we never have arrays as value in this map. So this should be good.


var pipelineId = randomAlphaOfLength(5);
var value = randomAlphaOfLength(5);
var pipelineString = "{\"processors\": [{\"set\" : {\"field\": \"_field\", \"value\": \"" + value + "\"}}]}";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe create a variant of this test that creates a random deep map? I don't think PipelineConfiguration cares about that content being valid.

@danhermann
Copy link
Contributor Author

@elasticmachine update branch

@danhermann
Copy link
Contributor Author

Thanks, @martijnvg!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >enhancement Team:Data Management Meta label for data/management team v7.16.0 v8.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize performance of updating ingest pipelines when operation is a no-op
4 participants