Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix remove ingest processor handing ignore_missing parameter not correctly #10089

Merged
merged 10 commits into from
Sep 25, 2023

Conversation

gaobinlong
Copy link
Collaborator

@gaobinlong gaobinlong commented Sep 18, 2023

Description

Remove ingest processor does not handle ignore_missing parameter correctly, when the resolved filed path of template snippet like {{field}} is empty string and ignore_missing is true, the processor does not exit quietly, but throw an illegal_argument_exception, this should be a bug.

The main change of this PR:

  1. Fix the bug
  2. Add some yml test and unit test

Related Issues

#10066

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…ectly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

github-actions bot commented Sep 18, 2023

Compatibility status:

Checks if related components are compatible with change 2042c87

Incompatible components

Incompatible components: [https://github.com/opensearch-project/k-nn.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/neural-search.git]

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Gao Binlong <gbinlong@amazon.com>
@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

@github-actions
Copy link
Contributor

Gradle Check (Jenkins) Run Completed with:

  • RESULT: UNSTABLE ❕
  • TEST FAILURES:
      1 org.opensearch.remotestore.RemoteStoreStatsIT.testStatsOnRemoteStoreRestore

@codecov
Copy link

codecov bot commented Sep 24, 2023

Codecov Report

Merging #10089 (2042c87) into main (1dde018) will decrease coverage by 0.20%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main   #10089      +/-   ##
============================================
- Coverage     71.22%   71.02%   -0.20%     
+ Complexity    58240    58138     -102     
============================================
  Files          4828     4828              
  Lines        274419   274423       +4     
  Branches      39989    39991       +2     
============================================
- Hits         195443   194905     -538     
- Misses        62574    63195     +621     
+ Partials      16402    16323      -79     
Files Changed Coverage Δ
.../org/opensearch/ingest/common/RemoveProcessor.java 96.96% <100.00%> (+7.31%) ⬆️

... and 454 files with indirect coverage changes

@msfroh msfroh merged commit c44337e into opensearch-project:main Sep 25, 2023
21 checks passed
@reta reta added the backport 2.x Backport to 2.x branch label Sep 25, 2023
opensearch-trigger-bot bot pushed a commit that referenced this pull request Sep 25, 2023
…ectly (#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
(cherry picked from commit c44337e)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
reta pushed a commit that referenced this pull request Sep 25, 2023
…ectly (#10089) (#10218)

* Fix remove ingest processor handing ignore_missing parameter not correctly



* modify change log



* Fix test failure and optimize some code



* Minor change



* Modify test code



* Split a test method to two



---------


(cherry picked from commit c44337e)

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
brusic pushed a commit to brusic/OpenSearch that referenced this pull request Sep 25, 2023
…ectly (opensearch-project#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Ivan Brusic <ivan.brusic@flocksafety.com>
vikasvb90 pushed a commit to vikasvb90/OpenSearch that referenced this pull request Oct 10, 2023
…ectly (opensearch-project#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
shiv0408 pushed a commit to Gaurav614/OpenSearch that referenced this pull request Apr 25, 2024
…ectly (opensearch-project#10089)

* Fix remove ingest processor handing ignore_missing parameter not correctly

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* modify change log

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Fix test failure and optimize some code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Minor change

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Modify test code

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

* Split a test method to two

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

---------

Signed-off-by: Gao Binlong <gbinlong@amazon.com>
Signed-off-by: Shivansh Arora <hishiv@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport 2.x Backport to 2.x branch
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants