[DOC] Add drop processor #5767

vagimeli · 2023-12-01T21:17:27Z

Description

Adds drop processor documentation to close content gap

Issues Resolved

#proce

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli · 2023-12-01T21:19:19Z

Hi @macohen & @msfroh, Will you review, or direct me to the appropriate reviewer, for the ingest processor documentation? We're working through content gaps and need dev reviewers to vet technical accuracy. Thank you.

META issue for reference #4647

vagimeli · 2023-12-21T20:44:36Z

@hdhalter I've requested several tech reviews via OpenSearch community Slack channel and AWS OpenSearch Slack channel. No response to date. Who is the SDM for ingest processors for OpenSearch core? The Data Prepper team told me that this isn't their lane.

msfroh · 2023-12-21T20:59:48Z

I can take a look at this once I'm back home on Tuesday.

vagimeli · 2023-12-21T22:49:28Z

I can take a look at this once I'm back home on Tuesday.

@msfroh You've been been more than helpful with tech reviewing the other three ingest processors. We're trying to load balance the content gap reviews, and I don't want you to get caught exclusively on this and the remaining 9-10 ingest processors. Please don't prioritize this while you're OOO. Let us see if we can find someone to share the load. I'll keep you updated. Thank you again for all your help and reviews :)

_ingest-pipelines/processors/drop.md

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli · 2024-02-15T21:12:54Z

@gaobinlong Please review this processor documentation at your availability. Will you provide examples relevant to OpenSearch users? Thank you for your support!

gaobinlong · 2024-02-26T06:32:19Z

_ingest-pipelines/processors/drop.md

+The following is the syntax for the `drop` processor:
+
+```json
+{


{
"drop": {
"if": "ctx.foo == 'bar'"
}
}

gaobinlong · 2024-02-26T06:59:46Z

_ingest-pipelines/processors/drop.md

+{
+  "user_info": "Sensitive information including credit card"
+}
+```


The following response shows that the no operation happened, the document with id 1 was not indexed:

{ "_index": "testindex1", "_id": "1", "_version": -3, "result": "noop", "_shards": { "total": 0, "successful": 0, "failed": 0 } }

gaobinlong · 2024-02-26T07:02:08Z

_ingest-pipelines/processors/date.md

Should this file be removed?

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli · 2024-02-26T22:06:43Z

@gaobinlong Thank you for the examples. I've added them to the doc. Will you approve as the tech reviewer? Thank you.

gaobinlong · 2024-02-27T02:28:07Z

_ingest-pipelines/processors/date.md

Oh my mad, this file shouldn't be removed, I just saw this change was not related to the drop processor so I thought it shouldn't be in this PR, actually it was a small change for the date processor, so please add this file back, thanks!

This reverts commit 73296f5.

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

gaobinlong

Looks good to me.

natebower

@vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

_ingest-pipelines/processors/drop.md

natebower · 2024-03-07T12:39:37Z

_ingest-pipelines/processors/drop.md

+
+Follow these steps to use the processor in a pipeline.
+
+**Step 1: Create a pipeline.**


Global: I believe that these do not take periods.

_ingest-pipelines/processors/drop.md

natebower · 2024-03-07T12:43:09Z

_ingest-pipelines/processors/drop.md

+```
+{% include copy-curl.html %}
+
+The following response confirms that the document with ID `1` was not indexed: 


"with the ID 1" or "with the ID of 1"?

natebower

LGTM with noted comments/changes.

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

* Add drop processor doc to address content gap Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Address tech review feedback Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Address tech review changes Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Delete _ingest-pipelines/processors/date.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Revert "Delete _ingest-pipelines/processors/date.md" This reverts commit 73296f5. * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update drop.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com> (cherry picked from commit b38ba75) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add drop processor doc to address content gap * Address tech review feedback * Address tech review changes * Delete _ingest-pipelines/processors/date.md * Revert "Delete _ingest-pipelines/processors/date.md" This reverts commit 73296f5. * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update _ingest-pipelines/processors/drop.md * Update drop.md --------- (cherry picked from commit b38ba75) Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

* Add drop processor doc to address content gap Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Address tech review feedback Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Address tech review changes Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Delete _ingest-pipelines/processors/date.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Revert "Delete _ingest-pipelines/processors/date.md" This reverts commit 73296f5. * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update _ingest-pipelines/processors/drop.md Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> * Update drop.md Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com> --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Nathan Bower <nbower@amazon.com>

Add drop processor doc to address content gap

20c5fc6

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli requested review from AMoo-Miki, Naarcha-AWS, ananzh, hdhalter, kolchfa-aws and natebower as code owners December 1, 2023 21:17

Merge branch 'main' into drop-processor

cc54f0b

vagimeli mentioned this pull request Dec 1, 2023

[META DOC] Ingest Processors Content Gaps #4647

Closed

45 tasks

vagimeli self-assigned this Dec 15, 2023

vagimeli added Needs SME Tech review PR: Tech review in progress Content gap and removed Needs SME labels Dec 15, 2023

msfroh reviewed Dec 29, 2023

View reviewed changes

_ingest-pipelines/processors/drop.md Outdated Show resolved Hide resolved

Address tech review feedback

6d0febd

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli added Doc review PR: Doc review in progress and removed Blocked Tech review PR: Tech review in progress labels Jan 10, 2024

vagimeli changed the title ~~[DOC] Add drop processor doc to address content gap~~ [DOC] Add drop processor Feb 15, 2024

gaobinlong reviewed Feb 26, 2024

View reviewed changes

Address tech review changes

35a2053

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli requested review from dlvenable and stephen-crawford as code owners February 26, 2024 21:29

Delete _ingest-pipelines/processors/date.md

73296f5

Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

gaobinlong reviewed Feb 27, 2024

View reviewed changes

vagimeli added 4 commits February 27, 2024 12:45

Revert "Delete _ingest-pipelines/processors/date.md"

b572c4d

This reverts commit 73296f5.

Merge branch 'main' into drop-processor

fb049e8

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Merge branch 'main' into drop-processor

a3650f6

Merge branch 'main' into drop-processor

0efa50e

gaobinlong approved these changes Mar 1, 2024

View reviewed changes

natebower reviewed Mar 7, 2024

View reviewed changes

natebower approved these changes Mar 7, 2024

View reviewed changes

vagimeli and others added 6 commits March 7, 2024 09:36

Update _ingest-pipelines/processors/drop.md

b6f7902

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _ingest-pipelines/processors/drop.md

8ee3c95

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _ingest-pipelines/processors/drop.md

0046644

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _ingest-pipelines/processors/drop.md

0ac9c1d

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _ingest-pipelines/processors/drop.md

8081f65

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update drop.md

4477666

Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli added 3 - Done backport 2.12 PR: Backport label for 2.12 and removed Doc review PR: Doc review in progress labels Mar 7, 2024

Merge branch 'main' into drop-processor

e996085

vagimeli merged commit b38ba75 into main Mar 7, 2024

vagimeli deleted the drop-processor branch March 7, 2024 16:45

opensearch-trigger-bot bot mentioned this pull request Mar 7, 2024

[Backport 2.12] [DOC] Add drop processor #6627

Merged


		Follow these steps to use the processor in a pipeline.

		Step 1: Create a pipeline.

Uh oh!

[DOC] Add drop processor #5767

[DOC] Add drop processor #5767

Uh oh!

Conversation

vagimeli commented Dec 1, 2023

Description

Issues Resolved

Checklist

Uh oh!

vagimeli commented Dec 1, 2023

Uh oh!

vagimeli commented Dec 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

msfroh commented Dec 21, 2023

Uh oh!

vagimeli commented Dec 21, 2023

Uh oh!

Uh oh!

vagimeli commented Feb 15, 2024

Uh oh!

gaobinlong Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

gaobinlong Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

gaobinlong Feb 26, 2024

Choose a reason for hiding this comment

Uh oh!

vagimeli commented Feb 26, 2024

Uh oh!

gaobinlong Feb 27, 2024

Choose a reason for hiding this comment

Uh oh!

gaobinlong left a comment

Choose a reason for hiding this comment

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

natebower Mar 7, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

natebower Mar 7, 2024

Choose a reason for hiding this comment

Uh oh!

natebower left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vagimeli commented Dec 21, 2023 •

edited

Loading