Skip to content

Commit a9e67fc

Browse files
vagimeliretahdhalternatebower
committed
[DOC] Add kv processor content gap doc (#5781)
* Add kv processor content gap doc --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Nathan Bower <nbower@amazon.com>
1 parent 104dc2a commit a9e67fc

File tree

2 files changed

+150
-3
lines changed

2 files changed

+150
-3
lines changed

‎_ingest-pipelines/processors/kv.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
---
2+
layout: default
3+
title: KV
4+
parent: Ingest processors
5+
nav_order: 200
6+
redirect_from:
7+
- /api-reference/ingest-apis/processors/lowercase/
8+
---
9+
10+
# KV processor
11+
12+
The `kv` processor automatically extracts specific event fields or messages that are in a `key=value` format. This structured format organizes your data by grouping it together based on keys and values. It's helpful for analyzing, visualizing, and using data, such as user behavior analytics, performance optimizations, or security investigations.
13+
14+
## Example
15+
16+
The following is the syntax for the `kv` processor:
17+
18+
```json
19+
{
20+
"kv": {
21+
"field": "message",
22+
"field_split": " ",
23+
"value_split": " "
24+
}
25+
}
26+
```
27+
{% include copy-curl.html %}
28+
29+
## Configuration parameters
30+
31+
The following table lists the required and optional parameters for the `kv` processor.
32+
33+
| Parameter | Required/Optional | Description |
34+
`field` | Required | The name of the field containing the data to be parsed. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |
35+
`field_split` | Required | The regex pattern for key-value pair splitting. |
36+
`value_split` | Required | The regex pattern for splitting the key from the value within a key-value pair, for example, equal sign `=` or colon `:`.
37+
`exclude_keys` | Optional | The keys to exclude from the document. Default is `null`. |
38+
`include_keys` | Optional | The keys for filtering and inserting. Default is to include all keys. |
39+
`prefix` | Optional | The prefix to add to the extracted keys. Default is `null`. |
40+
`strip_brackets` | Optional | If set to `true`, strips brackets (`()`, `<>,` or `[]`) and quotes (`'` or `"`) from extracted values. Default is `false`.
41+
`trim_key` | Optional | The string of characters to trim from the extracted keys. |
42+
`trim value` | Optional | The string of characters to trim from the extracted values. |
43+
`description` | Optional | A brief description of the processor. |
44+
`if` | Optional | A condition for running the processor. |
45+
`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. |
46+
`on_failure` | Optional | A list of processors to run if the processor fails. |
47+
`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not contain the specified field. Default is `false`. |
48+
`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. |
49+
`target_field` | Optional | The name of the field in which to insert the extracted keys. Default is `null`. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |
50+
51+
## Using the processor
52+
53+
Follow these steps to use the processor in a pipeline.
54+
55+
**Step 1: Create a pipeline**
56+
57+
The following query creates a pipeline, named `kv-pipeline`, that uses the `kv` processor to extract the `message` field of a document:
58+
59+
```json
60+
PUT _ingest/pipeline/kv-pipeline
61+
{
62+
"description" : "Pipeline that extracts user profile data",
63+
"processors" : [
64+
{
65+
"kv" : {
66+
"field" : "message",
67+
"field_split": " ",
68+
"value_split": "="
69+
}
70+
}
71+
]
72+
}
73+
```
74+
{% include copy-curl.html %}
75+
76+
**Step 2 (Optional): Test the pipeline**
77+
78+
It is recommended that you test your pipeline before you ingest documents.
79+
{: .tip}
80+
81+
To test the pipeline, run the following query:
82+
83+
```json
84+
POST _ingest/pipeline/kv-pipeline/_simulate
85+
86+
```json
87+
{
88+
"docs": [
89+
{
90+
"_index": "testindex1",
91+
"_id": "1",
92+
"_source":{
93+
"message": "goodbye=everybody hello=world"
94+
}
95+
}
96+
]
97+
}
98+
```
99+
{% include copy-curl.html %}
100+
101+
**Response**
102+
103+
The following example response confirms that the pipeline is working as expected:
104+
105+
```json
106+
{
107+
"docs": [
108+
{
109+
"doc": {
110+
"_index": "testindex1",
111+
"_id": "1",
112+
"_source": {
113+
"hello": "world",
114+
"message": "goodbye=everybody hello=world",
115+
"goodbye": "everybody"
116+
},
117+
"_ingest": {
118+
"timestamp": "2023-12-06T09:59:21.823292Z"
119+
}
120+
}
121+
}
122+
]
123+
}
124+
```
125+
126+
**Step 3: Ingest a document**
127+
128+
The following query ingests a document into an index named `testindex1`:
129+
130+
```json
131+
PUT testindex1/_doc/1?pipeline=kv-pipeline
132+
133+
```json
134+
{
135+
"message": "goodbye=everybody hello=world"
136+
}
137+
```
138+
{% include copy-curl.html %}
139+
140+
**Step 4 (Optional): Retrieve the document**
141+
142+
To retrieve the document, run the following query:
143+
144+
```json
145+
GET testindex1/_doc/1
146+
```
147+
{% include copy-curl.html %}

‎_ingest-pipelines/processors/lowercase.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,15 +29,15 @@ The following is the syntax for the `lowercase` processor:
2929

3030
The following table lists the required and optional parameters for the `lowercase` processor.
3131

32-
| Name | Required | Description |
32+
| Parameter | Required | Description |
3333
|---|---|---|
34-
`field` | Required | The name of the field that contains the data to be converted. Supports template snippets. |
34+
`field` | Required | The name of the field containing the data to be converted. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |
3535
`description` | Optional | A brief description of the processor. |
3636
`if` | Optional | A condition for running this processor. |
3737
`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. |
3838
`on_failure` | Optional | A list of processors to run if the processor fails. |
3939
`ignore_missing` | Optional | Specifies whether the processor should ignore documents that do not have the specified field. Default is `false`. |
40-
`tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. |
40+
`tag` | Optional | An identifier tag for the processor. Useful for debugging in order to distinguish between processors of the same type. |
4141
`target_field` | Optional | The name of the field in which to store the parsed data. Default is `field`. By default, `field` is updated in place. |
4242

4343
## Using the processor

0 commit comments

Comments
 (0)