Skip to content

Testing of tool call usage by LLM augmented with the MCP #36

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

dennisfeng200
Copy link

@dennisfeng200 dennisfeng200 commented May 29, 2025

Purpose

  • adds integration tests that verify whether the intended tool functions are called when given sensible user messages.
  • e.g. if user asks "what kinds of projects are on AI Foundry Labs" we test if the list-projects tool is called.

Does this introduce a breaking change?

[ ] Yes
[x] No

Pull Request Type

What kind of change does this Pull Request introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[x] Other... Please describe: adds integration tests

How to Test

  • Get the code
git clone [repo-address]
cd [repo-name]
git checkout [branch-name]
uv sync --prerelease allow --group test
  • ensure you have the correct env variables. You need to fill out the following 3 env variables with an Azure OpenAI endpoint:
AZURE_OPENAI_ENDPOINT=<placeholder>
AZURE_OPENAI_API_VERSION="2025-03-01-preview"
AZURE_OPENAI_DEPLOYMENT="gpt-4o"
  • Test the code
cd tests
uv run --prerelease allow --group test pytest --runintegration

What to Check

Verify that the following are valid

  • all tests pass. Might take 5+ min because it's calling LLM API.

Other Information

@dennisfeng200 dennisfeng200 marked this pull request as ready for review May 30, 2025 21:10
@dem108
Copy link
Contributor

dem108 commented Jun 20, 2025

I had shared a few checkpoints offline - please follow up to update this PR. Thanks! @dennisfeng200 @tendau

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants