Skip to content

Conversation

@pyek-bot
Copy link
Collaborator

@pyek-bot pyek-bot commented Sep 24, 2025

Description

When an agent runs, it creates a new interaction. On successful execution, this interaction is updated with the final response. However, on agent failures, the interaction never gets updated. Therefore, this PR addresses that gap.

Example:
Before in case of agent failures, response field is never populated

{
    "memory_id": "Uw_ufZkBE5GN_tuwO2KT",
    "message_id": "Wg_yfZkBE5GN_tuwuGI9",
    "create_time": "2025-09-24T22:58:02.939633Z",
    "input": "give me index mapping of any index",
}

After

{
    "memory_id": "Uw_ufZkBE5GN_tuwO2KT",
    "message_id": "Wg_yfZkBE5GN_tuwuGI9",
    "create_time": "2025-09-24T22:58:02.939633Z",
    "input": "give me index mapping of any index",
    "response": "Agent execution failed: failed ot run"
}

Related Issues

Resolves #4197

Check List

  • New functionality includes testing.
  • New functionality has been documented.
  • API changes companion pull request created.
  • Commits are signed per the DCO using --signoff.
  • Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@pyek-bot pyek-bot marked this pull request as draft September 25, 2025 00:51
@pyek-bot pyek-bot marked this pull request as ready for review September 25, 2025 01:41
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 25, 2025 01:42 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 25, 2025 01:42 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 25, 2025 01:42 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 25, 2025 01:42 — with GitHub Actions Inactive
@codecov
Copy link

codecov bot commented Sep 25, 2025

Codecov Report

❌ Patch coverage is 82.85714% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.74%. Comparing base (4864f66) to head (5b8aa24).
⚠️ Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
...ch/ml/engine/algorithms/agent/MLAgentExecutor.java 82.85% 3 Missing and 3 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #4198      +/-   ##
============================================
+ Coverage     81.69%   81.74%   +0.04%     
- Complexity     9374     9384      +10     
============================================
  Files           796      796              
  Lines         40622    40652      +30     
  Branches       4515     4519       +4     
============================================
+ Hits          33186    33230      +44     
+ Misses         5490     5471      -19     
- Partials       1946     1951       +5     
Flag Coverage Δ
ml-commons 81.74% <82.85%> (+0.04%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 25, 2025 02:45 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 25, 2025 02:45 — with GitHub Actions Inactive
@pyek-bot
Copy link
Collaborator Author

CI failure is unrelated & flaky

130 tests completed, 1 failed, 11 skipped
Tests with failures:
 - org.opensearch.ml.rest.RestMLRAGSearchProcessorIT.testBM25WithBedrockConverseUsingLlmMessages

@ylwu-amzn
Copy link
Collaborator

Rebase with main branch ?

Signed-off-by: Pavan Yekbote <pybot@amazon.com>
Signed-off-by: Pavan Yekbote <pybot@amazon.com>
Signed-off-by: Pavan Yekbote <pybot@amazon.com>
Signed-off-by: Pavan Yekbote <pybot@amazon.com>
Signed-off-by: Pavan Yekbote <pybot@amazon.com>
Signed-off-by: Pavan Yekbote <pybot@amazon.com>
Signed-off-by: Pavan Yekbote <pybot@amazon.com>
@pyek-bot pyek-bot force-pushed the fix-message-error-message branch from 88b554c to 5b8aa24 Compare September 29, 2025 21:17
@pyek-bot
Copy link
Collaborator Author

Rebase with main branch ?

Rebased!

@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 29, 2025 21:19 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 29, 2025 21:19 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 29, 2025 21:19 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 29, 2025 21:19 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 29, 2025 22:23 — with GitHub Actions Inactive
@pyek-bot pyek-bot temporarily deployed to ml-commons-cicd-env September 29, 2025 22:23 — with GitHub Actions Inactive
Copy link
Collaborator

@mingshl mingshl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good start, but we just do a general error message now. later when we know the pattern of the errors. we better to do different error handling for different types of failures (validation, execution, timeout). That would be better for user to understand rather than throwing the raw exception message.

@pyek-bot
Copy link
Collaborator Author

Thanks @mingshl Agreed!

Another improvement I notice:

  • A lot of the agent runners handle same logic in their individual classes
  • This can be centralized my moving the changes to the AgentExecutor (like returning a standard result)

This PR is a start towards that as the error message handling is happening in a common place.

@pyek-bot pyek-bot merged commit 5a7f4cf into opensearch-project:main Sep 29, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE] Update message with failure message when agent execution fails

6 participants