Buildkite Collectors for Ruby

DEPRECATION NOTICE Versions prior to 2.1.x are unsupported and will not work after mid-2023. Please upgrade to the latest version.

Official Buildkite Test Engine collectors for Ruby test frameworks ✨

⚒ Supported test frameworks: RSpec, Minitest, Cucumber, and more coming soon.

📦 Supported CI systems: Buildkite, GitHub Actions, CircleCI, Codeship, and others via the BUILDKITE_ANALYTICS_* environment variables.

👉 Installing

Step 1

Create a test suite, and copy the API token that it gives you.

Add the buildkite-test_collector gem:

gem install buildkite-test_collector

Or add this to your Gemfile’s test group:

group :test do
  gem 'buildkite-test_collector'
end

Step 2

RSpec

Add the following code to your RSpec setup file:

# spec/spec_helper.rb
require 'buildkite/test_collector'
Buildkite::TestCollector.configure(hook: :rspec)

Run your tests locally:

BUILDKITE_ANALYTICS_TOKEN=xyz rspec

Minitest

Add the following code to your Minitest setup file:

# test/test_helper.rb
require 'buildkite/test_collector'
Buildkite::TestCollector.configure(hook: :minitest)

Run your tests locally:

BUILDKITE_ANALYTICS_TOKEN=xyz rake

Cucumber

Add the following code to your Cucumber setup file:

# features/support/env.rb
require 'buildkite/test_collector'
Buildkite::TestCollector.configure(hook: :cucumber)

Run your tests locally:

BUILDKITE_ANALYTICS_TOKEN=xyz cucumber

Step 3

Add the BUILDKITE_ANALYTICS_TOKEN secret to your CI, push your changes to a branch, and open a pull request 🎉

OpenTelemetry export (experimental)

RSpec suites can also send an OpenTelemetry trace per test execution to Buildkite, showing what each test did and where it spent its time. Each trace is rooted in a test.execution span naming the test, its file, and whether it passed.

This is still under development and everything here may change. It is off by default, so opt in when you configure the collector:

Buildkite::TestCollector.configure(hook: :rspec, otel_enabled: true)

If your suite already runs OpenTelemetry, we use your existing setup and your instrumentation as it is. An otel_instrumentations: [] selection is ignored with a warning in that path. If the suite doesn't run OpenTelemetry, we set one up and install all applicable instrumentation registered when the suite starts. The collector does not include instrumentation gems. Add and explicitly require each one you want to use:

# Gemfile
gem "opentelemetry-instrumentation-pg", require: false

# spec/spec_helper.rb
require "opentelemetry-instrumentation-pg"
require "buildkite/test_collector"

Buildkite::TestCollector.configure(hook: :rspec, otel_enabled: true)

Adding a gem to the Gemfile may auto-require it in applications that call Bundler.require, but that is not guaranteed. An explicit require is the recommended setup. To disable instrumentations and export only root test.execution spans, set otel_instrumentations: []. Any other value is reserved for a future release and disables span export with a warning, regardless of who owns the OpenTelemetry setup. See the OpenTelemetry guide for more.

Export needs Ruby 3.3 or newer, which is what the OpenTelemetry gems require. On older Rubies the option is accepted and does nothing.

Spans need BUILDKITE_ANALYTICS_TOKEN to be an agent OIDC token with the write_uploads scope, from buildkite-agent oidc request-token. A suite API token still uploads executions, but its spans are rejected.

Export failures never fail a test or block the normal Test Engine upload. See the OpenTelemetry guide for what you get and how it fits around an existing OpenTelemetry setup.

More information

For more use cases such as custom tags, annotations, and span tracking, please visit our official Ruby collector documentation for details.

⚒ Developing

After cloning the repository, install the dependencies:

bundle

And run the tests:

bundle exec rspec

Useful resources for developing collectors include the Buildkite Test Engine docs.

See DESIGN.md for an overview of the design of this gem.

👩‍💻 Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/buildkite/test-collector-ruby

🚀 Releasing

1. Bump the version in version.rb and run bundle to update the Gemfile.lock.
2. Update the CHANGELOG.md with your new version and a description of your changes.
3. Once your PR is merged to main git tag the merge commit and push:

git tag vX.X.X
git push origin vX.X.X

4. Visit the release pipeline to unblock it and confirm the new version is pushed to rubygems.org
5. Create a new release in github.

📜 MIT License

The gem is available as open source under the terms of the MIT License.