integration test suite

[Copilot]

Basic test infrastructure to run the compiled cli and check that the LLM call succeeds. Currently tests "list", and "run" (org/non-org).

These tests are meant to make sure that the happy path of integration tests works.

[pelikhan]

@copilot apply comments

• add a "build" target in makefile that run script/build

[pelikhan]

remove this step

[Copilot]

Removed the "Install gh CLI" step from the workflow. (78b652d)

[Copilot]

@copilot apply comments

• add a "build" target in makefile that run script/build

Added "build" target to Makefile that runs script/build. The target is now available and used in the GitHub workflow. (78b652d)

[pelikhan]

@copilot apply comments

• add org test as well

[Copilot]

@copilot apply comments

• add org test as well

Added organization tests for both run and eval commands. The new tests (TestIntegrationRunWithOrg and TestIntegrationEvalWithOrg) validate the --org flag functionality while using help commands to avoid expensive API calls in CI. (296a540)

[pelikhan]

@copilot build the executable and run the tests. make sure the integration test executable is NOT gh-models, it can be confusing.

[Copilot]

@copilot build the executable and run the tests. make sure the integration test executable is NOT gh-models, it can be confusing.

Built the executable and ran the tests successfully. Changed the integration test binary name from gh-models to gh-models-test to avoid confusion. The make integration target now builds both the main gh-models binary and a separate gh-models-test binary specifically for integration testing. (1754609)

[Copilot]

Pull Request Overview

This PR introduces an integration test suite to validate the functionality of the compiled CLI binary against live endpoints. The test infrastructure includes authentication handling, timeout management, and basic validation of command outputs.

• Adds integration tests for list and run commands with graceful authentication fallback
• Creates build and test automation through Makefile targets
• Implements CI/CD workflow for automated integration testing

Reviewed Changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
integration/integration_test.go	Core test implementation with binary execution and command validation
integration/go.mod	Go module definition for test dependencies
integration/README.md	Documentation for running and understanding integration tests
Makefile	Build and integration test automation targets
DEV.md	Updated development documentation to include integration testing
.github/workflows/integration.yml	CI workflow for automated integration testing

Comments suppressed due to low confidence (3)

integration/integration_test.go:83

• The model version 'openai/gpt-4.1' may not exist. Consider using a well-known stable model version like 'openai/gpt-4' or 'openai/gpt-3.5-turbo' to ensure test reliability.

	hasExpectedContent := strings.Contains(lowerOut, "openai/gpt-4.1")

integration/integration_test.go:90

• The model version 'openai/gpt-4.1-nano' may not exist. Consider using a verified model name to prevent test failures due to invalid model references.

	stdout, _, err := runCommand(t, "run", "openai/gpt-4.1-nano", "say 'pain' in french")

integration/integration_test.go:98

• The model version 'openai/gpt-4.1-nano' may not exist. Consider using a verified model name to prevent test failures due to invalid model references.

	stdout, _, err := runCommand(t, "run", "openai/gpt-4.1-nano", "say 'pain' in french", "--org", "github")

[sgoedecke]

Sure, I think this is a good idea, as long as we keep the request count low and only trigger this on pushes to main. Good choice of 4.1-nano, so we're not consuming the more-popular 4.1 rate limit.

[pelikhan]

Sure, I think this is a good idea, as long as we keep the request count low and only trigger this on pushes to main. Good choice of 4.1-nano, so we're not consuming the more-popular 4.1 rate limit.

yes I figured a run push:main is cheap since we rarely push in this branch.