Skip to content

Commit 7bf4b54

Browse files
shuangli-zxinyualpengjunquan-lzane-neo
authored
[FEATURE] Add Index Insight Feature (opensearch-project#4088)
* [WIP] Initial version of index insight cron job Signed-off-by: shuangli-z <shualee@amazon.com> * merge Signed-off-by: xinyual <xinyual@amazon.com> * add index insight restful api Signed-off-by: xinyual <xinyual@amazon.com> * fix next token Signed-off-by: xinyual <xinyual@amazon.com> * fix next token Signed-off-by: xinyual <xinyual@amazon.com> * [WIP] initial version of IndexInsightTask design Signed-off-by: shuangli-z <shualee@amazon.com> * add access controller Signed-off-by: xinyual <xinyual@amazon.com> * Parallelize LLM calls in FieldDescriptionTask by batching Signed-off-by: shuangli-z <shualee@amazon.com> * add waiting logic when prerequisite tasks are in GENERATING status Signed-off-by: shuangli-z <shualee@amazon.com> * remove MLIndexInsightType.ALL and Index Insight List in api since IndexInsightTask is now processed one at a time Signed-off-by: shuangli-z <shualee@amazon.com> * Change logic of sample fetching Signed-off-by: shuangli-z <shualee@amazon.com> * add put container Signed-off-by: xinyual <xinyual@amazon.com> * Implemented basic execute operation - TODO: Add prerequisite chain logic Signed-off-by: shuangli-z <shualee@amazon.com> * Add log related index check task for index insight Signed-off-by: shuangli-z <shualee@amazon.com> * update LogRelatedIndexCheckTask to match interface modifications Signed-off-by: shuangli-z <shualee@amazon.com> * Add logic to execute prerequisite task(s) first Signed-off-by: shuangli-z <shualee@amazon.com> * add put index isight container Signed-off-by: xinyual <xinyual@amazon.com> * Remove redundant task status checks in GetIndexInsightTransportAction Signed-off-by: shuangli-z <shualee@amazon.com> * add put/delete container API Signed-off-by: xinyual <xinyual@amazon.com> * refactor: remove deprecated implementation of index insight cron job Signed-off-by: shuangli-z <shualee@amazon.com> * merge latest code Signed-off-by: xinyual <xinyual@amazon.com> * pass tenant id and add agent execution logic Signed-off-by: xinyual <xinyual@amazon.com> * merge from main Signed-off-by: xinyual <xinyual@amazon.com> * add aggregation Signed-off-by: xinyual <xinyual@amazon.com> * change dsl Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * use agent for LLM calls and container for insights storage Signed-off-by: shuangli-z <shualee@amazon.com> * resolve statistical data processing bugs and other minor fixes Signed-off-by: shuangli-z <shualee@amazon.com> * refine statistical task Signed-off-by: xinyual <xinyual@amazon.com> * add license Signed-off-by: shuangli-z <shualee@amazon.com> * spotless updates Signed-off-by: shuangli-z <shualee@amazon.com> * remove uselessResponse Signed-off-by: xinyual <xinyual@amazon.com> * remove deprecated code Signed-off-by: shuangli-z <shualee@amazon.com> * use correct return status Signed-off-by: xinyual <xinyual@amazon.com> * set order for StatisticalDataTask result Signed-off-by: shuangli-z <shualee@amazon.com> * add UT Signed-off-by: xinyual <xinyual@amazon.com> * add failure cases Signed-off-by: xinyual <xinyual@amazon.com> * add UT for get index insight action Signed-off-by: xinyual <xinyual@amazon.com> * add UT for delete/put container Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * delete IndexDescriptionTask and modify parsing logic in FieldDescriptionTask Signed-off-by: shuangli-z <shualee@amazon.com> * add UT for response and request Signed-off-by: xinyual <xinyual@amazon.com> * add license Signed-off-by: xinyual <xinyual@amazon.com> * address PR comments in FieldDescriptionTask.java Signed-off-by: shuangli-z <shualee@amazon.com> * fix simple commit Signed-off-by: xinyual <xinyual@amazon.com> * optimize logic creating container Signed-off-by: xinyual <xinyual@amazon.com> * fix comment Signed-off-by: xinyual <xinyual@amazon.com> * Reorganize methods in IndexInsightTask and address PR comments Signed-off-by: shuangli-z <shualee@amazon.com> * apply spotless Signed-off-by: shuangli-z <shualee@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * modify prompt in FieldDescriptionTask to avoid specific examples Signed-off-by: shuangli-z <shualee@amazon.com> * add merge helper for statistical data task Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * use pre defined mapping Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * add Tool Signed-off-by: xinyual <xinyual@amazon.com> * add index insight tool Signed-off-by: xinyual <xinyual@amazon.com> * add UT for index insight and improve code structure Signed-off-by: shuangli-z <shualee@amazon.com> * add more ut Signed-off-by: xinyual <xinyual@amazon.com> * add LogRelatedIndexCheckTaskTests Signed-off-by: juqnuanp <junquanp@amazon.com> * revert useless change Signed-off-by: xinyual <xinyual@amazon.com> * add UT for IndexInsightTask interface and its implementations Signed-off-by: shuangli-z <shualee@amazon.com> * resolve comments Signed-off-by: xinyual <xinyual@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * address CR comments Signed-off-by: shuangli-z <shualee@amazon.com> * address CR comments Signed-off-by: shuangli-z <shualee@amazon.com> * fix UT Signed-off-by: shuangli-z <shualee@amazon.com> * address PR comments Signed-off-by: shuangli-z <shualee@amazon.com> * change container name keyword Signed-off-by: xinyual <xinyual@amazon.com> * add try catch exception Signed-off-by: xinyual <xinyual@amazon.com> * add more tests for LogRelatedIndexCheckTaskTests Signed-off-by: juqnuanp <junquanp@amazon.com> * add UT for LogRelatedIndexCheckTaskTests and refine other tests Signed-off-by: shuangli-z <shualee@amazon.com> * modify put api to put index insight config Signed-off-by: xinyual <xinyual@amazon.com> * modify put api to put index insight config Signed-off-by: xinyual <xinyual@amazon.com> * add IndexInsightTestHelper to reduce code duplication Signed-off-by: shuangli-z <shualee@amazon.com> * modifying API Signed-off-by: xinyual <xinyual@amazon.com> * add get config API Signed-off-by: xinyual <xinyual@amazon.com> * rename Signed-off-by: xinyual <xinyual@amazon.com> * add MLIndexInsightType.ALL Signed-off-by: shuangli-z <shualee@amazon.com> * add license Signed-off-by: xinyual <xinyual@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * add UT Signed-off-by: xinyual <xinyual@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * add UT for MLIndexInsightType.ALL Signed-off-by: shuangli-z <shualee@amazon.com> * fix IT Signed-off-by: xinyual <xinyual@amazon.com> * add UT Signed-off-by: xinyual <xinyual@amazon.com> * add IT Signed-off-by: xinyual <xinyual@amazon.com> * rename Signed-off-by: xinyual <xinyual@amazon.com> * refine code structure for MLIndexInsightType.ALL execution Signed-off-by: shuangli-z <shualee@amazon.com> * filter useless in statistical and field description Signed-off-by: xinyual <xinyual@amazon.com> * fix statistical threshold Signed-off-by: xinyual <xinyual@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * fix IT Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * set to very low result Signed-off-by: xinyual <xinyual@amazon.com> * remove useless IT Signed-off-by: xinyual <xinyual@amazon.com> * add script build Signed-off-by: xinyual <xinyual@amazon.com> * add pattern match Signed-off-by: xinyual <xinyual@amazon.com> * move storage to system index Signed-off-by: xinyual <xinyual@amazon.com> * refine pattern match logic Signed-off-by: shuangli-z <shualee@amazon.com> * use sdkclient to search/get/put index insight Signed-off-by: xinyual <xinyual@amazon.com> * fix UT Signed-off-by: shuangli-z <shualee@amazon.com> * address PR comments Signed-off-by: shuangli-z <shualee@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * fix IT Signed-off-by: xinyual <xinyual@amazon.com> * fixLogic Signed-off-by: xinyual <xinyual@amazon.com> * add and fix UT Signed-off-by: shuangli-z <shualee@amazon.com> * fix comment for zna Signed-off-by: xinyual <xinyual@amazon.com> * remove useless cide Signed-off-by: xinyual <xinyual@amazon.com> * fix comment 2 Signed-off-by: xinyual <xinyual@amazon.com> * fix error message logic Signed-off-by: xinyual <xinyual@amazon.com> * add one more ut Signed-off-by: xinyual <xinyual@amazon.com> * add ut for tool Signed-off-by: xinyual <xinyual@amazon.com> * remove deprecated variable storageIndex Signed-off-by: shuangli-z <shualee@amazon.com> * add filter by LLM Signed-off-by: xinyual <xinyual@amazon.com> * apply spotless Signed-off-by: xinyual <xinyual@amazon.com> * change to empty return Signed-off-by: xinyual <xinyual@amazon.com> * change remaining number Signed-off-by: xinyual <xinyual@amazon.com> * filter by sample Signed-off-by: xinyual <xinyual@amazon.com> * skip filter when column number is very few Signed-off-by: xinyual <xinyual@amazon.com> * make pattern static Signed-off-by: xinyual <xinyual@amazon.com> * address PR comments Signed-off-by: shuangli-z <shualee@amazon.com> * address PR comments: add AbstractIndexInsightTask and remove IndexInsightUtils Signed-off-by: shuangli-z <shualee@amazon.com> * add permission check Signed-off-by: xinyual <xinyual@amazon.com> * minor fixes Signed-off-by: shuangli-z <shualee@amazon.com> * minor fixes Signed-off-by: shuangli-z <shualee@amazon.com> * Update plugin/src/main/java/org/opensearch/ml/action/IndexInsight/GetIndexInsightTransportAction.java Co-authored-by: zane-neo <zaniu@amazon.com> Signed-off-by: Xinyuan Lu <xinyual@amazon.com> * Update ml-algorithms/src/main/java/org/opensearch/ml/engine/tools/IndexInsightTool.java Co-authored-by: zane-neo <zaniu@amazon.com> Signed-off-by: Xinyuan Lu <xinyual@amazon.com> * fix comment for zaniu Signed-off-by: xinyual <xinyual@amazon.com> * address PR comments and minor fixes Signed-off-by: shuangli-z <shualee@amazon.com> * add permission check Signed-off-by: xinyual <xinyual@amazon.com> * minor fixes Signed-off-by: shuangli-z <shualee@amazon.com> * fix comment for zaniu Signed-off-by: xinyual <xinyual@amazon.com> * minor fixes Signed-off-by: shuangli-z <shualee@amazon.com> * fix bugs caused by rebasing Signed-off-by: shuangli-z <shualee@amazon.com> * fix pr comment Signed-off-by: xinyual <xinyual@amazon.com> * change pattern match logic Signed-off-by: shuangli-z <shualee@amazon.com> * address PR comments Signed-off-by: shuangli-z <shualee@amazon.com> * add validation logic to check if index pattern exist Signed-off-by: shuangli-z <shualee@amazon.com> * fix UT Signed-off-by: xinyual <xinyual@amazon.com> * add a cluster setting and close it by default Signed-off-by: xinyual <xinyual@amazon.com> * Delete store logic for StatisticalDataTask when pattern matched Signed-off-by: shuangli-z <shualee@amazon.com> * fix error message Signed-off-by: xinyual <xinyual@amazon.com> --------- Signed-off-by: shuangli-z <shualee@amazon.com> Signed-off-by: xinyual <xinyual@amazon.com> Signed-off-by: juqnuanp <junquanp@amazon.com> Signed-off-by: Xinyuan Lu <xinyual@amazon.com> Co-authored-by: xinyual <xinyual@amazon.com> Co-authored-by: pengjunquan-l <pjq750830166@gmail.com> Co-authored-by: juqnuanp <junquanp@amazon.com> Co-authored-by: zane-neo <zaniu@amazon.com>
1 parent 467a8ee commit 7bf4b54

File tree

67 files changed

+8480
-7
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

67 files changed

+8480
-7
lines changed

‎common/src/main/java/org/opensearch/ml/common/CommonValue.java‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,9 @@ public class CommonValue {
3333
public static String HOT_BOX_TYPE = "hot";
3434
// warm node
3535
public static String WARM_BOX_TYPE = "warm";
36+
public static final String ML_INDEX_INSIGHT_CONFIG_INDEX = ".plugins-ml-index-insight-config";
37+
public static final String ML_INDEX_INSIGHT_STORAGE_INDEX = ".plugins-ml-index-insight-storage";
38+
3639
public static final String ML_MODEL_GROUP_INDEX = ".plugins-ml-model-group";
3740
public static final String ML_MODEL_INDEX = ".plugins-ml-model";
3841
public static final String ML_TASK_INDEX = ".plugins-ml-task";
@@ -68,6 +71,8 @@ public class CommonValue {
6871
public static final String ML_MCP_SESSION_MANAGEMENT_INDEX_MAPPING_PATH = "index-mappings/ml_mcp_session_management.json";
6972
public static final String ML_MCP_TOOLS_INDEX_MAPPING_PATH = "index-mappings/ml_mcp_tools.json";
7073
public static final String ML_JOBS_INDEX_MAPPING_PATH = "index-mappings/ml_jobs.json";
74+
public static final String ML_INDEX_INSIGHT_CONFIG_INDEX_MAPPING_PATH = "index-mappings/ml_index_insight_config.json";
75+
public static final String ML_INDEX_INSIGHT_STORAGE_INDEX_MAPPING_PATH = "index-mappings/ml_index_insight_storage.json";
7176

7277
// Calculate Versions independently of OpenSearch core version
7378
public static final Version VERSION_2_11_0 = Version.fromString("2.11.0");
@@ -111,4 +116,9 @@ public class CommonValue {
111116

112117
// TOOL Constants
113118
public static final String TOOL_INPUT_SCHEMA_FIELD = "input_schema";
119+
120+
public static final String INDEX_INSIGHT_AGENT_NAME = "os_index_insight_agent";
121+
public static final long INDEX_INSIGHT_GENERATING_TIMEOUT = 3 * 60 * 1000;
122+
public static final long INDEX_INSIGHT_UPDATE_INTERVAL = 24 * 60 * 60 * 1000;
123+
114124
}

‎common/src/main/java/org/opensearch/ml/common/MLIndex.java‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,10 @@
1515
import static org.opensearch.ml.common.CommonValue.ML_CONNECTOR_INDEX_MAPPING_PATH;
1616
import static org.opensearch.ml.common.CommonValue.ML_CONTROLLER_INDEX;
1717
import static org.opensearch.ml.common.CommonValue.ML_CONTROLLER_INDEX_MAPPING_PATH;
18+
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_CONFIG_INDEX;
19+
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_CONFIG_INDEX_MAPPING_PATH;
20+
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_STORAGE_INDEX;
21+
import static org.opensearch.ml.common.CommonValue.ML_INDEX_INSIGHT_STORAGE_INDEX_MAPPING_PATH;
1822
import static org.opensearch.ml.common.CommonValue.ML_JOBS_INDEX;
1923
import static org.opensearch.ml.common.CommonValue.ML_JOBS_INDEX_MAPPING_PATH;
2024
import static org.opensearch.ml.common.CommonValue.ML_MCP_SESSION_MANAGEMENT_INDEX_MAPPING_PATH;
@@ -50,7 +54,9 @@ public enum MLIndex {
5054
MEMORY_CONTAINER(ML_MEMORY_CONTAINER_INDEX, false, ML_MEMORY_CONTAINER_INDEX_MAPPING_PATH),
5155
MCP_SESSION_MANAGEMENT(MCP_SESSION_MANAGEMENT_INDEX, false, ML_MCP_SESSION_MANAGEMENT_INDEX_MAPPING_PATH),
5256
MCP_TOOLS(MCP_TOOLS_INDEX, false, ML_MCP_TOOLS_INDEX_MAPPING_PATH),
53-
JOBS(ML_JOBS_INDEX, false, ML_JOBS_INDEX_MAPPING_PATH);
57+
JOBS(ML_JOBS_INDEX, false, ML_JOBS_INDEX_MAPPING_PATH),
58+
INDEX_INSIGHT_CONFIG(ML_INDEX_INSIGHT_CONFIG_INDEX, false, ML_INDEX_INSIGHT_CONFIG_INDEX_MAPPING_PATH),
59+
INDEX_INSIGHT_STORAGE(ML_INDEX_INSIGHT_STORAGE_INDEX, false, ML_INDEX_INSIGHT_STORAGE_INDEX_MAPPING_PATH);
5460

5561
private final String indexName;
5662
// whether we use an alias for the index

0 commit comments

Comments
 (0)