[BUG] Duplicate trace record

[Hailong-am]

What is the bug?
Run MLChatAgent and check all the trace records in .plugins-ml-memory-message index, there will have two trace records with same content for final answer.

An example response, you can see trace with number 6 and 7 are duplicate

[
    {
        "interactionId": "x95uXo0BGVYn19hhheIJ",
        "input": "can you show me the visualization of sales",
        "output": "Now I know how to respond",
        "createTime": "2024-01-31T07:31:57.832292374Z",
        "origin": null,
        "traceNumber": 1
    },
    {
        "interactionId": "yN5uXo0BGVYn19hhheIX",
        "input": "sales",
        "output": "Title,Id\n[eCommerce] Sales by Category,37cc8650-b882-11e8-a6d9-e546fe2bba5f\n[eCommerce] Sales by Gender,ed8436b0-b88b-11e8-a6d9-e546fe2bba5f\n[eCommerce] Average Sales Price,4b3ec120-b892-11e8-a6d9-e546fe2bba5f\n",
        "createTime": "2024-01-31T07:31:57.841167178Z",
        "origin": "VisualizationTool",
        "traceNumber": 2
    },
    {
        "interactionId": "zN5uXo0BGVYn19hhi-L7",
        "input": "can you show me the visualization of sales",
        "output": "Based on the visualizations provided, it seems the [eCommerce] Sales by Category visualization would be most relevant to understand sales breakdown by category.",
        "createTime": "2024-01-31T07:31:59.610137456Z",
        "origin": null,
        "traceNumber": 3
    },
    {
        "interactionId": "zd5uXo0BGVYn19hhi-L_",
        "input": "[eCommerce] Sales by Category",
        "output": "Title,Id\n[eCommerce] Sales by Category,37cc8650-b882-11e8-a6d9-e546fe2bba5f\n[eCommerce] Sales by Gender,ed8436b0-b88b-11e8-a6d9-e546fe2bba5f\n[eCommerce] Average Sales Price,4b3ec120-b892-11e8-a6d9-e546fe2bba5f\n",
        "createTime": "2024-01-31T07:31:59.614597716Z",
        "origin": "VisualizationTool",
        "traceNumber": 4
    },
    {
        "interactionId": "0N5uXo0BGVYn19hhl-Ia",
        "input": "can you show me the visualization of sales",
        "output": "Now I know the final answer",
        "createTime": "2024-01-31T07:32:02.45740304Z",
        "origin": null,
        "traceNumber": 5
    },
    {
        "interactionId": "z95uXo0BGVYn19hhl-Ia",
        "input": "can you show me the visualization of sales",
        "output": "To see the visualization of sales you had requested, one of the available visualizations was shown containing metrics like sales by category, sales by gender, and average sales price for the day.",
        "createTime": "2024-01-31T07:32:02.457582962Z",
        "origin": null,
        "traceNumber": 6
    },
    {
        "interactionId": "zt5uXo0BGVYn19hhl-Ia",
        "input": "can you show me the visualization of sales",
        "output": "To see the visualization of sales you had requested, one of the available visualizations was shown containing metrics like sales by category, sales by gender, and average sales price for the day.",
        "createTime": "2024-01-31T07:32:02.457707056Z",
        "origin": null,
        "traceNumber": 7
    }
]

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

What is the expected behavior?
should not have duplicate trace records

What is your host/environment?

• OS: [e.g. iOS]
• Version [e.g. 22]
• Plugins

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?
Looks like this code are duplicate
https://github.com/opensearch-project/ml-commons/blob/main/ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/agent/MLChatAgentRunner.java#L377-L405

[Hailong-am]

close it due to #1972