Support output filter, unify tool parameter handling and improve Sear…

[ylwu-amzn]

…chIndexTool output parsing

Description

Fix issues when build Agentic RAG with automatic index selection tutorial

1. In the tutorial, the Chat Agent uses AgentTool to call another Flow Agent. The flow agent generates OpenSearch query and search index. The Flow Agent returns extra info, but we only need the query response part. So we need tool output filter.

2. The Flow Agent will pass the generated OpenSearch query to SearchIndexTool to search data. But the escape is wrong. The query is already valid json. If we run escapeJson(outputResponse) again, the query json string become invalid.

3. In this tutorial, we use one LLM which has multiple placeholders like _chat_history, tool_configs. The root Chat Agent is Chat Agent, which has chat history and tool configs in parameters. When Chat Agent run the Flow Agent through a tool call, it will also pass these parameters to Flow Agent. Since Flow Agent use same LLM, these parameter will cause wrong result in Flow Agent. So we need to support required parameter on tool level to filter out only the required parameters.

4. SearchIndexTool always returns the search hits part. But for an aggregation query, the search hits part is empty. Add a new parameter to return the full response.
   For example

GET opensearch_dashboards_sample_data_flights/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "OriginCountry": "CA"
          }
        },
        {
          "term": {
            "DestCountry": "US"
          }
        }
      ]
    }
  },
  "size": 0,
  "aggs": {
    "flight_count": {
      "value_count": {
        "field": "FlightNum"
      }
    }
  }
}

Response

{
  "took": 14,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 94,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "flight_count": {
      "value": 94
    }
  }
}

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[pyek-bot]

LGTM

[ylwu-amzn]

Security token expried

RestQueryPlanningToolIT > testAgentWithQueryPlanningTool_DefaultPrompt FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://[::1]:34149], URI [/_plugins/_ml/agents/7Lbne5gBcIyZu0YtP25o/_execute], status line [HTTP/1.1 403 Forbidden]
    {"status":403,"error":{"type":"OpenSearchStatusException","reason":"System Error","details":"Error from remote service: {\"message\":\"The security token included in the request is expired\"}"}}
        at __randomizedtesting.SeedInfo.seed([5052B41FA5D28C5C:9AA7FDE762437DE5]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:501)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:384)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:359)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:199)
        at app//org.opensearch.ml.utils.TestHelper.makeRequest(TestHelper.java:172)
        at app//org.opensearch.ml.rest.RestQueryPlanningToolIT.executeAgent(RestQueryPlanningToolIT.java:181)