generated from amazon-archives/__template_Apache-2.0
-
Couldn't load subscription status.
- Fork 621
[DOC] Add html strip processor documentation #5984
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
16 commits
Select commit
Hold shift + click to select a range
fabcb3b
Add html strip processor documentation
vagimeli 41a2bdf
Add html strip processor documentation
vagimeli 3cb41ec
Merge branch 'main' into html-strip
vagimeli de535f7
Add examples
vagimeli bb91c7e
Merge branch 'main' into html-strip
vagimeli b32aaf8
Merge branch 'main' into html-strip
vagimeli c76e12f
Copy edits
vagimeli 5803bd3
Update _ingest-pipelines/processors/html-strip.md
vagimeli 764c597
Update _ingest-pipelines/processors/html-strip.md
vagimeli 0d6356a
Update _ingest-pipelines/processors/html-strip.md
vagimeli 4d06d4d
Update _ingest-pipelines/processors/html-strip.md
vagimeli a2ab21e
Update _ingest-pipelines/processors/html-strip.md
vagimeli 2f7f17c
Update _ingest-pipelines/processors/html-strip.md
vagimeli f5a3d62
Update _ingest-pipelines/processors/html-strip.md
vagimeli 8fd346d
Update _ingest-pipelines/processors/html-strip.md
vagimeli ad46155
Merge branch 'main' into html-strip
vagimeli File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| --- | ||
| layout: default | ||
| title: HTML strip | ||
| parent: Ingest processors | ||
| nav_order: 140 | ||
| --- | ||
|
|
||
| # HTML strip processor | ||
|
|
||
| The `html_strip` processor removes HTML tags from string fields in incoming documents. This processor is useful when indexing data from webpages or other sources that may contain HTML markup. HTML tags are replaced with newline characters (`\n`). | ||
|
|
||
| The following is the syntax for the `html_strip` processor: | ||
|
|
||
| ```json | ||
| { | ||
| "html_strip": { | ||
| "field": "webpage" | ||
| } | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| ## Configuration parameters | ||
|
|
||
| The following table lists the required and optional parameters for the `html_strip` processor. | ||
|
|
||
| Parameter | Required/Optional | Description | | ||
| |-----------|-----------|-----------| | ||
| `field` | Required | The string field from which to remove HTML tags. | ||
| `target_field` | Optional | The field that receives the plain text version after stripping HTML tags. If not specified, then the field is updated in-place. | ||
| `ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified field. Default is `false`. | ||
| `description` | Optional | A description of the processor's purpose or configuration. | ||
| `if` | Optional | Specifies to conditionally execute the processor. | ||
| `ignore_failure` | Optional | Specifies to ignore processor failures. See [Handling pipeline failures]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/). | ||
| `on_failure` | Optional | Specifies a list of processors to run if the processor fails during execution. These processors are executed in the order they are specified. See [Handling pipeline failures]({{site.url}}{{site.baseurl}}/ingest-pipelines/pipeline-failures/). | ||
| `tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. | ||
|
|
||
| ## Using the processor | ||
|
|
||
| Follow these steps to use the processor in a pipeline. | ||
|
|
||
| ### Step 1: Create a pipeline | ||
|
|
||
| The following query creates a pipeline named `strip-html-pipeline` that uses the `html_strip` processor to remove HTML tags from the description field and store the processed value in a new field named `cleaned_description`: | ||
|
|
||
| ```json | ||
| PUT _ingest/pipeline/strip-html-pipeline | ||
| { | ||
| "description": "A pipeline to strip HTML from description field", | ||
vagimeli marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "processors": [ | ||
| { | ||
| "html_strip": { | ||
| "field": "description", | ||
| "target_field": "cleaned_description" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| ### Step 2 (Optional): Test the pipeline | ||
|
|
||
| It is recommended that you test your pipeline before you ingest documents. | ||
| {: .tip} | ||
|
|
||
| To test the pipeline, run the following query: | ||
|
|
||
| ```json | ||
| POST _ingest/pipeline/strip-html-pipeline/_simulate | ||
| { | ||
| "docs": [ | ||
| { | ||
| "_source": { | ||
| "description": "This is a <b>test</b> description with <i>some</i> HTML tags." | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| #### Response | ||
|
|
||
| The following example response confirms that the pipeline is working as expected: | ||
|
|
||
| ```json | ||
| { | ||
| "docs": [ | ||
| { | ||
| "doc": { | ||
| "_index": "_index", | ||
| "_id": "_id", | ||
| "_source": { | ||
| "description": "This is a <b>test</b> description with <i>some</i> HTML tags.", | ||
| "cleaned_description": "This is a test description with some HTML tags." | ||
| }, | ||
| "_ingest": { | ||
| "timestamp": "2024-05-22T21:46:11.227974965Z" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| ### Step 3: Ingest a document | ||
|
|
||
| The following query ingests a document into an index named `products`: | ||
|
|
||
| ```json | ||
| PUT products/_doc/1?pipeline=strip-html-pipeline | ||
| { | ||
| "name": "Product 1", | ||
| "description": "This is a <b>test</b> product with <i>some</i> HTML tags." | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| #### Response | ||
|
|
||
| The response shows that the request has indexed the document into the index `products` and will index all documents with the `description` field containing HTML tags while storing the plain text version in the `cleaned_description` field: | ||
|
|
||
| ```json | ||
| { | ||
| "_index": "products", | ||
| "_id": "1", | ||
| "_version": 1, | ||
| "result": "created", | ||
| "_shards": { | ||
| "total": 2, | ||
| "successful": 1, | ||
| "failed": 0 | ||
| }, | ||
| "_seq_no": 0, | ||
| "_primary_term": 1 | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| ### Step 4 (Optional): Retrieve the document | ||
|
|
||
| To retrieve the document, run the following query: | ||
|
|
||
| ```json | ||
| GET products/_doc/1 | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| #### Response | ||
|
|
||
| The response includes both the original `description` field and the `cleaned_description` field with HTML tags removed: | ||
|
|
||
| ```json | ||
| { | ||
| "_index": "products", | ||
| "_id": "1", | ||
| "_version": 1, | ||
| "_seq_no": 0, | ||
| "_primary_term": 1, | ||
| "found": true, | ||
| "_source": { | ||
| "cleaned_description": "This is a test product with some HTML tags.", | ||
| "name": "Product 1", | ||
| "description": "This is a <b>test</b> product with <i>some</i> HTML tags." | ||
| } | ||
| } | ||
| ``` | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.