Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Guide] Index Lifecycle #349

Closed
Tracked by #355
nhtruong opened this issue Apr 4, 2023 · 2 comments · Fixed by #362
Closed
Tracked by #355

[Guide] Index Lifecycle #349

nhtruong opened this issue Apr 4, 2023 · 2 comments · Fixed by #362
Labels
CCI enhancement New feature or request good first issue Good for newcomers

Comments

@nhtruong
Copy link
Contributor

nhtruong commented Apr 4, 2023

Create guides/index_lifecycle.md similar to the Ruby guide

You must assure that:

  • All code-blocks, when run from start to finish in order they appear in the guide, do not throw any errors. If you want to demonstrate that certain statements will throw errors, they must be wrapped in a try/catch and print out the expected error.
  • All code-blocks are properly marked with the language it's written in (i.e. ```ruby)
  • The instructions, though mostly identical to the Ruby guide, are suitable for this repo.
@nhtruong nhtruong added enhancement New feature or request good first issue Good for newcomers untriaged Need triage CCI labels Apr 4, 2023
@nhtruong nhtruong mentioned this issue Apr 4, 2023
6 tasks
@nhtruong nhtruong removed the untriaged Need triage label Apr 4, 2023
@ReinGrad
Copy link

ReinGrad commented Apr 4, 2023

for the "index_lifecycle.md" guide in the OpenSearch Python client library (opensearch-py) repository:

Index Lifecycle Management with OpenSearch-Py

Index Lifecycle Management (ILM) is a feature in OpenSearch that allows you to manage the lifecycle of your indices, including actions such as rollover, deletion, and other operations based on certain conditions. This guide will walk you through how to use ILM with the OpenSearch-Py library, which is a Python client for interacting with OpenSearch.

Prerequisites

Before you start, make sure you have the following:

  • OpenSearch-Py library installed (you can install it using pip: pip install opensearch-py)
  • Access to an OpenSearch cluster

Setting up an Index Template

To use ILM, you first need to set up an index template that defines the properties of your indices, including the ILM policies. Here's an example of how you can create an index template using OpenSearch-Py:

from opensearch import OpenSearch
from opensearch.exceptions import OpenSearchException

# Connect to your OpenSearch cluster
opensearch = OpenSearch(["localhost"], verify_certs=True)

# Define the index template
index_template = {
    "index_patterns": ["my_index-*"],  # Specify the pattern for matching index names
    "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
    },
    "mappings": {
        "_source": {
            "enabled": False
        },
        "properties": {
            "timestamp": {
                "type": "date"
            },
            "message": {
                "type": "text"
            }
        }
    },
    "aliases": {
        "my_alias": {}  # Specify any aliases you want to create for the index
    },
    "lifecycle": {
        "name": "my_policy",  # Specify the name of the ILM policy to be associated with the index
        "rollover_alias": "my_alias"  # Specify the alias to be used for rollover
    }
}

try:
    # Create the index template
    opensearch.indices.put_index_template(name="my_template", body=index_template)
    print("Index template created successfully")
except OpenSearchException as e:
    print(f"Failed to create index template: {e}")

we are creating an index template named "my_template" with a rollover alias named "my_alias". We have also specified an ILM policy named "my_policy", which will be associated with the index that we will define in the next section.

Configuring the Index Lifecycle Policy

An AnIM policy is a set of rules that define the actions to be taken with an index when certain conditions are met. you can create an instant messaging policy using OpenSearch-Py:

from opensearch import OpenSearch
from opensearch.exceptions import OpenSearchException

# Connect to your OpenSearch cluster
opensearch = OpenSearch(["localhost"], verify_certs=True)

# Define the ILM policy
ilm_policy = {
    "policy": {
        "phases": {
            "hot": {  # Define the actions to be taken in the "hot" phase
                "actions": {
                    "rollover": {
                        "max_size": "50gb"  # Define the condition for rollover based on index size
                    }
                }
            },
            "delete": {  # Define the actions to be taken in the "delete" phase
                "min_age": "30d
                "actions": {
                    "delete": {}  # Define the action to delete the index
                }
            }
        }
    }
}

try:
    # Create the ILM policy
    opensearch.ilm.put_lifecycle_policy(policy="my_policy", body=ilm_policy)
    print("ILM policy created successfully")
except OpenSearchException as e:
    print(f"Failed to create ILM 
```policy: {e}")

we create an instant messaging policy name "my_policy" with two names: "hot" and "delete". At the "shot" stage, we performed an action to transfer the index when its size reaches 50 GB. At the "delete" stage, we defined an action to delete the index after it had been in the "delete" field for 30 days.


### still in development
*Applying the Index Template and ILM Policy to an Index. 
*Testing the FILM Actions.
*the index state

@Nicksqain
Copy link
Contributor

@nhtruong I have made a PR to solve this issue #362

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CCI enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants