Skip to content

ci: add integration test workflow #201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 11, 2025

Conversation

dbschmigelski
Copy link
Member

Description

This PR implements secure integration testing via GitHub Actions. It adds a workflow that safely executes integration tests against LLMs while maintaining security through proper authentication and access controls.

Regarding security concerns and their mitigations, we have implemented several layers of protection. For external contributor access, workflows require explicit approval and PRs from forks do not automatically receive access to secrets or OIDC. Credential protection is achieved through short-lived credentials using GitHub Actions OIDC, with secrets managed through secure mechanisms and access strictly limited to specific resources and actions.

Code execution is controlled through the use of pull_request_target for trusted execution context, requiring either owner status or an explicit 'approved-for-integ-test' label which can only be set by users with write access. Resource access is carefully managed by executing in a dedicated test environment with limited permissions scoped specifically to testing needs, and all access is monitored and controlled.

Related Issues

Several follow up issues will be created

  1. Until a central credit card is in place the following providers will not be covered: ANTHROPIC, OPENAI, LLAMAAPI

  2. There seems to be an issue with the streamable http integration test. The test passes locally, and streamable http has been confirmed to work by integrators. As we debug this test we need to uncover if there is a streamable http issue, a platform issue of MacOS vs Linux, or something else.

Documentation PR

N/A

Type of Change

New feature

Testing

Integration testing has been implemented through GitHub Actions workflow. The tests execute against both Bedrock and third-party LLM providers. We have updated the region configurations to us-east-1 across several test files including test_bedrock_guardrails.py, test_model_litellm.py, and test_mcp_client.py.

As the proposed workflow is apull_request_target it will only be run after being merged. So for testing, in the forked repo you can see a successful run https://github.com/dbschmigelski/sdk-python/actions/runs/15562697157/job/43827517559. Note, the IAM role is slightly different since it is targeting a personal account.

** Note, the STRANDS_INTEG_TEST_ROLE secret will only be set after this PR is approved. **

  • hatch fmt --linter
  • hatch fmt --formatter
  • hatch test --all
  • Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

Checklist

  • I have read the CONTRIBUTING document
  • I have added tests that prove my fix is effective or my feature works
  • [n/a] I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@dbschmigelski dbschmigelski changed the title Dea/ci integ tests ci: add integration test workflow Jun 10, 2025
@dbschmigelski dbschmigelski requested a review from pgrayy June 10, 2025 17:35
@dbschmigelski dbschmigelski requested a review from zastrowm June 11, 2025 15:51
@dbschmigelski
Copy link
Member Author

dbschmigelski commented Jun 11, 2025

Budget has been created in internal test aws account and STRANDS_INTEG_TEST_ROLE secret has been set.

@dbschmigelski dbschmigelski merged commit 4b44410 into strands-agents:main Jun 11, 2025
20 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants