Skip to content

Combine 'Index mgmt' with 'Index mgmt in Kibana' #2010

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ $$$ec-restrictions-apis-kibana$$$
* {{es}} plugins, are not enabled by default for security purposes. Reach out to support if you would like to enable {{es}} plugins support on your account.
* Some {{es}} plugins do not apply to {{ecloud}}. For example, you won’t ever need to change discovery, as {{ecloud}} handles how nodes discover one another.
% * In {{es}} 5.0 and later, site plugins are no longer supported. This change does not affect the site plugins {{ecloud}} might provide out of the box, such as Kopf or Head, since these site plugins are serviced by our proxies and not {{es}} itself.
% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) if you want more detailed information or perform index management actions.
% * In {{es}} 5.0 and later, site plugins such as Kopf and Paramedic are no longer provided. We recommend that you use our [cluster performance metrics](../../monitor/stack-monitoring.md), [X-Pack monitoring features](../../monitor/stack-monitoring.md) and Kibana’s (6.3+) [Index Management UI](/manage-data/data-store/index-basics.md#manage-indices) if you want more detailed information or perform index management actions.


## Watcher [ec-restrictions-watcher]
Expand Down
2 changes: 1 addition & 1 deletion explore-analyze/alerts-cases/watcher/watcher-ui.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ To use Watcher in {{kib}}, you must have the built-in `kibana_admin` role and ei
To manage roles, go to the **Roles** management page, or use the [role APIs](https://www.elastic.co/docs/api/doc/kibana/group/endpoint-roles). Watches are shared between all users with the same role.

::::{note}
If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](../../../manage-data/lifecycle/index-lifecycle-management/index-management-in-kibana.md) for detailed information.
If you are creating a threshold watch, you must also have the `view_index_metadata` index privilege. See [Index management](/manage-data/data-store/index-basics.md#index-management) for detailed information.
::::

## Create a threshold alert [watcher-create-threshold-alert]
Expand Down
2 changes: 1 addition & 1 deletion manage-data/data-store/data-streams/logs-data-stream.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ PUT _index_template/my-index-template

After the index template is created, new indices that use the template will be configured as a logs data stream. You can start indexing data and [using the data stream](use-data-stream.md).

You can also set the index mode and adjust other template settings in [the Elastic UI](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md).
You can also set the index mode and adjust other template settings in [the Elastic UI](/manage-data/data-store/index-basics.md#index-management-manage-index-templates).


## Synthetic source [logsdb-synthetic-source]
Expand Down
201 changes: 192 additions & 9 deletions manage-data/data-store/index-basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,23 +68,75 @@ Each index has a [mapping](/manage-data/data-store/mapping.md) or schema for how

## Index management

Elastic's index management features are an easy, convenient way to manage your cluster's indices, data streams, index templates, and enrich policies. Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible.
Elastic's **Index Management** features are an easy, convenient way to manage your clusters indices, [data streams](/manage-data/lifecycle/data-stream.md), [templates](/manage-data/data-store/templates.md), and [enrich policies](/manage-data/ingest/transform-enrich/data-enrichment.md). Practicing good index management ensures your data is stored correctly and in the most cost-effective way possible.

Go to **{{project-settings}} → {{manage-app}} → {{index-manage-app}}**.
To use these features:
* On {{serverless-full}}, go to **Project settings** > **Management** > **Index Management**.
* On {{stack}}, go to **Stack Management** > **Index Management**.

### Required permissions [index-mgm-req-permissions]
```{applies_to}
stack: ga
```

If you use {{es}} {{security-features}}, the following [security privileges](/deploy-manage/users-roles/cluster-or-deployment-auth/elasticsearch-privileges.md) are required:

* The `monitor` cluster privilege to access {{kib}}'s **Index Management** features.
* The `view_index_metadata` and `manage` index privileges to view a data stream or index’s data.
* The `manage_index_templates` cluster privilege to manage index templates.

To add these privileges, go to **Stack Management > Security > Roles** or use the [Create or update roles API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-security-put-role).

### Manage indices

Investigate your indices and perform operations from the **Indices** view.

:::::{tab-set}
:group: stack-serverless

::::{tab-item} {{serverless-short}}
:sync: serverless

:::{image} /manage-data/images/serverless-index-management-indices.png
:alt: Index Management indices
:screenshot:
:::

* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md).

* To show details and perform operations, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices).

* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze).

* To drill down into the index overivew, [mappings](/manage-data/data-store/mapping.md), and [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index.

::::

::::{tab-item} {{stack}}
:sync: stack

:::{image} /manage-data/images/elasticsearch-reference-management_index_labels.png
:alt: Index Management UI
:screenshot:
:::

* Enable **Include hidden indices** to view the full set of indices, including backing indices for [data streams](/manage-data/data-store/data-streams.md).

* To show details and perform operations such as close, forcemerge, and flush, click the index name. To perform operations on multiple indices, select their checkboxes and then open the **Manage** menu. For more information on managing indices, refer to [Index APIs](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-indices).

* To filter the list of indices, use the search bar or click a badge. Badges indicate if an index is a [follower index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ccr-follow), a [rollup index](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-rollup-get-rollup-index-caps), or [frozen](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-unfreeze).
* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), and statistics, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index.
* To create new indices, use the **Create index** wizard.

You can also use the drop-down menus to filter the list by the [index lifecycle](/manage-data/lifecycle/index-lifecycle-management/index-lifecycle.md) status or phase.

* To drill down into the index [mappings](/manage-data/data-store/mapping.md), [settings](elasticsearch://reference/elasticsearch/index-settings/index.md), statistics, and lifecycle details, click an index name. From this view, you can navigate to **Discover** to further explore the documents in the index.

:::{image} /manage-data/images/elasticsearch-reference-management_index_details.png
:alt: Index Management UI
:screenshot:
:::

::::
:::::

### Manage data streams

Expand All @@ -101,8 +153,8 @@ In {{es-serverless}}, indices matching the `logs-*-*` pattern use the logsDB ind

* To view information about the stream's backing indices, click the number in the **Indices** column.
* A value in the **Data retention** column indicates that the data stream is managed by a data stream lifecycle policy. This value is the time period for which your data is guaranteed to be stored. Data older than this period can be deleted by {{es}} at a later time.
* To modify the data retention value, select an index, open the **Manage** menu, and click **Edit data retention**.
* To view more information about a data stream, such as its generation or its current index lifecycle policy, click the stream's name. From this view, you can navigate to **Discover** to further explore data within the data stream.
* To modify the data retention value, select an index, open the **Manage** menu, and click **Edit data retention**. On {{stack}}, this action is only available if your data stream is not managed by an ILM policy.
* To view more information about a data stream including it's lifecycle settings, click the stream's name.

### Manage index templates [index-management-manage-index-templates]

Expand All @@ -120,6 +172,123 @@ Create, edit, clone, and delete your index templates in the **Index Templates**
* Values in the **Content** column indicate whether a template contains index mappings, settings, and aliases.
* To create new index templates, use the **Create template** wizard.

#### Try it: Create an index template [_try_it_create_an_index_template]

In this tutorial, you’ll create an index template and use it to configure two new indices.

##### Step 1. Add a name and index pattern

1. In the **Index Templates** view, open the **Create template** wizard.

:::{image} /manage-data/images/elasticsearch-reference-management_index_create_wizard.png
:alt: Create wizard
:screenshot:
:::

2. In the **Name** field, enter `my-index-template`.
3. Set **Index pattern** to `my-index-*` so the template matches any index with that index pattern.
4. Leave **Data Stream**, **Priority**, **Version**, and **_meta field** blank or as-is.

##### Step 2. Add settings, mappings, and aliases

When creating an index template, you can define settings, mappings, and aliases directly in the template or include them through one or more component templates.

A [component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) is a type of [template](/manage-data/data-store/templates.md) used as a building block for constructing index templates. {{kib}} displays badges indicating whether a component template contains mappings (**M**), index settings (**S**), aliases (**A**), or a combination of the three.

1. Add component templates to your index template.

Component templates are optional. For this tutorial, do not add any component templates.

:::{image} /manage-data/images/elasticsearch-reference-management_index_component_template.png
:alt: Component templates page
:screenshot:
:::

2. Define index settings directly in the index template. When used in conjunction with component templates, settings defined directly in the index template override any conflicting settings from the associated component templates.

This step is optional. For this tutorial, leave this section blank.
3. Define mappings directly in the index template. When used in conjunction with component templates, these mappings override any conflicting definitions from the associated component templates.

Define a mapping that contains an [object](elasticsearch://reference/elasticsearch/mapping-reference/object.md) field named `geo` with a child [`geo_point`](elasticsearch://reference/elasticsearch/mapping-reference/geo-point.md) field named `coordinates`:

:::{image} /manage-data/images/elasticsearch-reference-management-index-templates-mappings.png
:alt: Mapped fields page
:screenshot:
:::

Alternatively, you can click the **Load JSON** link and define the mapping as JSON:

```js
{
"properties": {
"geo": {
"properties": {
"coordinates": {
"type": "geo_point"
}
}
}
}
}
```

You can create additional mapping configurations in the **Dynamic templates** and **Advanced options** tabs. For this tutorial, do not create any additional mappings.

4. Define an alias named `my-index`:

```js
{
"my-index": {}
}
```

5. On the review page, check the summary. If everything looks right, click **Create template**.

##### Step 3. Create new indices

You’re now ready to create new indices using your index template.

1. Index the following documents to create two indices: `my-index-000001` and `my-index-000002`.

```console
POST /my-index-000001/_doc
{
"@timestamp": "2019-05-18T15:57:27.541Z",
"ip": "225.44.217.191",
"extension": "jpg",
"response": "200",
"geo": {
"coordinates": {
"lat": 38.53146222,
"lon": -121.7864906
}
},
"url": "https://media-for-the-masses.theacademyofperformingartsandscience.org/uploads/charles-fullerton.jpg"
}

POST /my-index-000002/_doc
{
"@timestamp": "2019-05-20T03:44:20.844Z",
"ip": "198.247.165.49",
"extension": "php",
"response": "200",
"geo": {
"coordinates": {
"lat": 37.13189556,
"lon": -76.4929875
}
},
"memory": 241720,
"url": "https://theacademyofperformingartsandscience.org/people/type:astronauts/name:laurel-b-clark/profile"
}
```

2. Use the [get index API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-get) to view the configurations for the new indices. The indices were configured using the index template you created earlier.

```console
GET /my-index-000001,my-index-000002
```

### Manage component templates

[Component templates](/manage-data/data-store/templates.md) are reusable building blocks that configure mappings, settings, and aliases.
Expand All @@ -136,9 +305,15 @@ Create, edit, clone, and delete your component templates in the **Component Temp

### Manage enrich policies

An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add the right enrich data to the right incoming documents.
An [enrich policy](/manage-data/ingest/transform-enrich/data-enrichment.md#enrich-policy) is a set of configuration options used to add data from your existing indices to incoming documents during ingest. An enrich policy contains:

Add data from your existing indices to incoming documents using the **Enrich Policies** view.
* The policy type that determines how the policy matches the enrich data to incoming documents
* The source indices that store enrich data as documents
* The fields from the source indices used to match incoming documents
* The enrich fields containing enrich data from the source indices that you want to add to incoming documents
* An optional [query](elasticsearch://reference/query-languages/query-dsl/query-dsl-match-all-query.md).

Use the **Enrich Policies** view to add data from your existing indices to incoming documents during ingest.

:::{image} /manage-data/images/serverless-management-enrich-policies.png
:alt: Enrich policies
Expand All @@ -149,6 +324,14 @@ Add data from your existing indices to incoming documents using the **Enrich Pol
* To perform operations, click the policy name or use the buttons in the **Actions** column.
* To create new policies, use the **Create enrich policy** wizard.

You must execute a new enrich policy before you can use it with an enrich processor. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents.
You must execute a new enrich policy before you can use it with an enrich processor or {{esql}} query. When executed, an enrich policy uses enrich data from the policy's source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents.

When creating an enrich policy, the UI walks you through the configuration setup and selecting the fields. Before you can use the policy with an enrich processor or {{esql}} query, you must execute the policy.

When executed, an enrich policy uses enrich data from the policy’s source indices to create a streamlined system index called the enrich index. The policy uses this index to match and enrich incoming documents.

Check out these examples:

* [Example: Enrich your data based on geolocation](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-geolocation.md)
* [Example: Enrich your data based on exact values](/manage-data/ingest/transform-enrich/example-enrich-data-based-on-exact-values.md)
* [Example: Enrich your data by matching a value to a range](/manage-data/ingest/transform-enrich/example-enrich-data-by-matching-value-to-range.md)
2 changes: 1 addition & 1 deletion manage-data/data-store/templates.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ If you use {{fleet}} or {{agent}}, assign your index templates a priority lower

## Create index template [create-index-templates]

Use the [index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) and [put component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) APIs to create and update index templates. You can also [manage index templates](../lifecycle/index-lifecycle-management/index-management-in-kibana.md) from Stack Management in {{kib}}.
Use the [put index template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template) and [put component template](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-cluster-put-component-template) APIs to create and update index templates. You can also [manage index templates](/manage-data/data-store/index-basics.md#index-management) from Stack Management in {{kib}}.

The following requests create two component templates.

Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified manage-data/images/serverless-index-management-indices.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion manage-data/ingest/transform-enrich/ingest-pipelines.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@ $$$pipeline-custom-logs-index-template$$$

2. Create an [index template](../../data-store/templates.md) that includes your pipeline in the [`index.default_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-default-pipeline) or [`index.final_pipeline`](elasticsearch://reference/elasticsearch/index-settings/index-modules.md#index-final-pipeline) index setting. Ensure the template is [data stream enabled](../../data-store/data-streams/set-up-data-stream.md#create-index-template). The template’s index pattern should match `logs-<dataset-name>-*`.

You can create this template using {{kib}}'s [**Index Management**](../../lifecycle/index-lifecycle-management/index-management-in-kibana.md#manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template).
You can create this template using {{kib}}'s [**Index Management**][index management](/manage-data/data-store/index-basics.md#index-management-manage-index-templates) feature or the [create index template API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-put-index-template).

For example, the following request creates a template matching `logs-my_app-*`. The template uses a component template that contains the `index.default_pipeline` index setting.

Expand Down
Loading
Loading