Skip to content

Conversation

@ofalvai
Copy link
Collaborator

@ofalvai ofalvai commented Aug 15, 2025

Checklist

Version

Requires a MAJOR/MINOR/PATCH version update

Context

Integration tests at the moment are in a folder called _tests in order to exclude them from the Go package when other packages import it as a dependency. This has two drawbacks, all because of the fact that we disable Go module features:

  1. All integration tests are in a single folder. We can't break them up and have cross-package imports (such as shared test helpers) because we disabled Go module lookup explicitly.
  2. All IDE support (except for syntax highlighting) are disabled and we catch obvious errors (e.g. unused imports, incorrect types) only in CI.

Changes

The main magic trick is this go.mod file:

module github.com/bitrise-io/bitrise/v2/integrationtests

go 1.23.0

// Magic trick that allows us to exclude integration tests from the main Go package
// (other repos consume the main package as a Go lib) and keep full Go LSP support for this package.
replace github.com/bitrise-io/bitrise/v2 => ../

require (
	github.com/bitrise-io/bitrise/v2 v2.0.0
)

Additional changes:

  • Fix test result parsing for failed test runs. It wasn't working when the go test command had a non-zero exit code.
  • Add VSCode settings.json for a nice default experience

Investigation details

Verification that this PR doesn't delete or disable tests accidentally:

Reference build from master: 101 integration tests

image

PR build: 101 integration tests

image

Setting -p 1 to serially execute tests unfortunately regresses test run time from ~7 minutes to ~11 minutes, but we can easily shard tests in the future by the Go modules this PR creates.

Decisions

Out of scope for this PR in order to keep the diff small:

  • shard tests based on test package
  • getting rid of $INTEGRATION_TEST_BINARY_PATH...it's going to be more complicated

@ofalvai ofalvai force-pushed the reorganize-integration-tests branch 4 times, most recently from 5d8661f to 8d3992a Compare August 15, 2025 13:51
@ofalvai ofalvai force-pushed the reorganize-integration-tests branch 7 times, most recently from 683b4ce to 254f6ba Compare August 18, 2025 10:02
@ofalvai ofalvai force-pushed the reorganize-integration-tests branch from 254f6ba to b5d5452 Compare August 18, 2025 10:12
@@ -0,0 +1,4 @@
{
"go.testTags": "linux_and_mac",
"go.buildTags": "linux_and_mac"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without these tags, the Go LSP doesn't handle the integration test files (since most of them are explicitly tagged with this)

@@ -1,82 +0,0 @@
{
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused file, couldn't find any references to it

@@ -1,61 +0,0 @@
format_version: 1.3.0
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unused file too

@@ -1,23 +0,0 @@
//go:build linux_and_mac
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not deleted, just moved

@ofalvai ofalvai marked this pull request as ready for review August 18, 2025 10:25
@ofalvai ofalvai merged commit 3f4bd63 into master Aug 19, 2025
8 checks passed
@ofalvai ofalvai deleted the reorganize-integration-tests branch August 19, 2025 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants