generated from amazon-archives/__template_Apache-2.0
-
Couldn't load subscription status.
- Fork 621
[DOC] Add new documentation for date index name processor #5398
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
27 commits
Select commit
Hold shift + click to select a range
9a75346
Add new processor documentation
vagimeli 5b086b8
Add new processor documentation
vagimeli db4159b
Add new processor documentation
vagimeli 538e1fe
Merge branch 'main' into date-index-processor
vagimeli 7527382
Merge branch 'main' into date-index-processor
vagimeli 2e517e3
Merge branch 'main' into date-index-processor
vagimeli f02dd15
Merge branch 'main' into date-index-processor
vagimeli 318e1f7
Address tech review feedback
vagimeli 353349d
Address tech review feedback
vagimeli af6429e
Merge branch 'main' into date-index-processor
vagimeli 6ad1a3f
Revise intro and examples to use date math index name use cases
vagimeli 06e7db4
Add date math example
vagimeli 7ac6b41
Update _ingest-pipelines/processors/date-index-name.md
vagimeli cfb9a44
Merge branch 'main' into date-index-processor
vagimeli e044ea5
Add date math example
vagimeli b85b7d4
Add date math example
vagimeli 83a032c
Update processor example to use date math index expression
vagimeli 1e62b04
Merge branch 'main' into date-index-processor
vagimeli f5dbdc5
Update processor example to use date math index expression
vagimeli 150dc27
Update _ingest-pipelines/processors/date-index-name.md
vagimeli d33706a
Update _ingest-pipelines/processors/date-index-name.md
vagimeli 68579e0
Update _ingest-pipelines/processors/date-index-name.md
vagimeli 4ea66b0
Update _ingest-pipelines/processors/date-index-name.md
vagimeli 90111c9
Update _ingest-pipelines/processors/date-index-name.md
vagimeli e393bdb
Update _ingest-pipelines/processors/date-index-name.md
vagimeli f8284df
Merge branch 'main' into date-index-processor
vagimeli bba5595
Fix dead link
vagimeli File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| --- | ||
| layout: default | ||
| title: Date index name | ||
| parent: Ingest processors | ||
| nav_order: 55 | ||
| --- | ||
|
|
||
| # Date index name | ||
|
|
||
| The `date_index_name` processor is used to point documents to the correct time-based index based on the date or timestamp field within the document. The processor sets the `_index` metadata field to a [date math]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/date/#date-math) index name expression. Then the processor fetches the date or timestamp from the `field` field in the document being processed and formats it into a date math index name expression. The extracted date, `index_name_prefix` value, and `date_rounding` value are then combined to create the date math index expression. For example, if the `field` field contains the value `2023-10-30T12:43:29.000Z` and `index_name_prefix` is set to `week_index-` and `date_rounding` is set to `w`, then the date math index name expression is `week_index-2023-10-30`. You can use the `date_formats` field to specify how the date in the date math index expression should be formatted. | ||
|
|
||
| The following is the syntax for the `date_index_name` processor: | ||
|
|
||
| ```json | ||
| { | ||
| "date_index_name": { | ||
| "field": "your_date_field or your_timestamp_field", | ||
| "date_rounding": "rounding_value" | ||
| } | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| ## Configuration parameters | ||
|
|
||
| The following table lists the required and optional parameters for the `date_index_name` processor. | ||
|
|
||
| Parameter | Required/Optional | Description | | ||
| |-----------|-----------|-----------| | ||
| `field` | Required | The date or timestamp field in the incoming document. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | | ||
| `date_rounding` | Required | The rounded date format within the index name . Valid values are `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute), and `s` (second). | | ||
| `date_formats` | Optional | An array of date formats used to parse the date or timestamp field. Valid options include a java time pattern or one of the following formats: ISO8601, UNIX, UNIX_MS, or TAI64N. Default is `yyyy-MM-dd'T'HH:mm:ss.SSSXX`. | | ||
| `index_name_format` | Optional | The date format. Default is `yyyy-MM-dd`. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | | ||
| `index_name_prefix` | Optional | The index name prefix to append before the date. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | ||
| `description` | Optional | A brief description of the processor. | | ||
| `if` | Optional | A condition for running this processor. | | ||
| `ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. | | ||
| `locale` | `locale` | Optional | The locale to use when parsing the month name and week day of the date. Default is `ENGLISH`. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). | | ||
| `on_failure` | Optional | A list of processors to run if the processor fails. | | ||
| `tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. | | ||
| `timezone` | Optional | The time zone to use when parsing the date. Default is `UTC`. | | ||
|
|
||
| ## Using the processor | ||
|
|
||
| Follow these steps to use the processor in a pipeline. | ||
|
|
||
| **Step 1: Create a pipeline.** | ||
|
|
||
| The following query creates a pipeline, named `date-index-name1`, that uses the `date_index_name` processor to index logs into monthly indexes: | ||
|
|
||
| ```json | ||
| PUT /_ingest/pipeline/date-index-name1 | ||
| { | ||
| "description": "Create weekly index pipeline", | ||
| "processors": [ | ||
| { | ||
| "date_index_name": { | ||
| "field": "date_field", | ||
| "index_name_prefix": "week_index-", | ||
| "date_rounding": "w", | ||
| "date_formats": ["YYYY-MM-DD"] | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| **Step 2 (Optional): Test the pipeline.** | ||
|
|
||
| It is recommended that you test your pipeline before you ingest documents. | ||
| {: .tip} | ||
|
|
||
| To test the pipeline, run the following query: | ||
|
|
||
| ```json | ||
| POST _ingest/pipeline/date-index-name1/_simulate | ||
| { | ||
| "docs": [ | ||
| { | ||
| "_index": "testindex1", | ||
| "_id": "1", | ||
| "_source": { | ||
| "date_field": "2023-10-30" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| #### Response | ||
|
|
||
| The following example response confirms that the pipeline is working as expected: | ||
|
|
||
| ```json | ||
| { | ||
| "docs": [ | ||
| { | ||
| "doc": { | ||
| "_index": "<week_index-{2023-10-01||/w{yyyy-MM-dd|UTC}}>", | ||
| "_id": "1", | ||
| "_source": { | ||
| "date_field": "2023-10-30" | ||
| }, | ||
| "_ingest": { | ||
| "timestamp": "2023-11-13T18:23:10.408593092Z" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| **Step 3: Ingest a document.** | ||
|
|
||
| The following query ingests a document into an index named `testindex1`: | ||
|
|
||
| ```json | ||
| PUT testindex1/_doc/1?pipeline=date-index-name1 | ||
| { | ||
| "date_field": "2023-10-30" | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| #### Response | ||
|
|
||
| The request indexes the document into the index `week_index-2023-10-23` and will index all documents with a timestamp within that week into the same index because the pipeline rounds by week. | ||
|
|
||
| ```json | ||
| { | ||
| "_index": "week_index-2023-10-30", | ||
| "_id": "1", | ||
| "_version": 4, | ||
| "result": "updated", | ||
| "_shards": { | ||
| "total": 2, | ||
| "successful": 1, | ||
| "failed": 0 | ||
| }, | ||
| "_seq_no": 3, | ||
| "_primary_term": 1 | ||
| } | ||
| ``` | ||
|
|
||
| **Step 4 (Optional): Retrieve the document.** | ||
|
|
||
| To retrieve the document, run the following query: | ||
|
|
||
| ```json | ||
| GET week_index-2023-10-30/_doc/1 | ||
| ``` | ||
| {% include copy-curl.html %} | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.