Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(ci): refactor indiv jobs #432

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 40 additions & 17 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,38 @@ on:
branches:
- master

workflow_dispatch:

schedule:
- cron: 0 0 * * 4 # Midnight Wednesday

concurrency:
group: release-${{ github.ref }}
cancel-in-progress: true

jobs:
build-release:
name: build-release (${{ matrix.targets }})
runs-on: ${{ matrix.operating-system }}
strategy:
matrix:
include:
- operating-system: ubuntu-20.04
targets: x86_64-unknown-linux-gnu,x86_64-unknown-linux-musl,aarch64-unknown-linux-gnu,aarch64-unknown-linux-musl
targets: x86_64-unknown-linux-gnu
- operating-system: ubuntu-20.04
targets: x86_64-unknown-linux-musl
- operating-system: ubuntu-20.04
targets: aarch64-unknown-linux-gnu
- operating-system: ubuntu-20.04
targets: aarch64-unknown-linux-musl
- operating-system: windows-2019
targets: aarch64-pc-windows-msvc,x86_64-pc-windows-msvc
targets: x86_64-pc-windows-msvc
- operating-system: windows-2019
targets: aarch64-pc-windows-msvc
- operating-system: macos-12
targets: aarch64-apple-darwin,x86_64-apple-darwin
targets: x86_64-apple-darwin
- operating-system: macos-14
targets: aarch64-apple-darwin
fail-fast: false

env:
Expand All @@ -43,6 +59,7 @@ jobs:
uses: Swatinem/rust-cache@v2
with:
workspaces: rust
key: ${{ matrix.targets }}

- name: Set up QEMU
if: runner.os == 'Linux'
Expand Down Expand Up @@ -71,52 +88,58 @@ jobs:
id: cargo-flags
shell: bash
run: |
if [[ "${{ github.event_name }}" = "release" ]]; then
echo "flags=--release" >> "$GITHUB_OUTPUT"
else
echo "flags=" >> "$GITHUB_OUTPUT"
fi

echo "flags=--release" >> "$GITHUB_OUTPUT"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since release builds can be significantly slower, should this still be only selectively enabled?

Copy link
Member Author

@YOU54F YOU54F Jun 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so originally when I was able to build for multi targets with cross in a single run, but I actually run out of disk space on the gh runner because the debug build folder was so large (and also results in a hefty cache to store and retrieve).

this workflow is only for building the released artifacts, until this PR where it does the most basic of smoke tests.

the actual build ffi workflow builds and tests the ffi lib cross plat (although not all of our targets, just x86_64 win/lin/mac)

so the rationale was if this is able to smoke test a full release wf before each target is actually released (so via the PR workflow), we actually know whether it will work when we push a release tag.

Maybe we could just run debug on the pr's, but run in release mode on master, so before someone does push a release tag, they have confidence it will just work (tm)

I feel like we could actually probably consolidate some of our workflows to avoid de-depulication, and maximise our caching potential. we have a pact rust workflow which tests everything, pact ffi workflow that builds the project with cmake (which we don't use in the release workflow), and c smoke tests on linux against.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we cache the CARGO_TARGET_DIR, we would likely get most of the speed up anyway. The compiler would not need to recompile most of the dependencies in the first place.


- name: Build verifier CLI
if: |
github.event_name == 'push' ||
github.event_name == 'pull_request' ||
github.event_name == 'workflow_dispatch' ||
startsWith(github.ref, 'refs/tags/pact_verifier_cli')
shell: bash
run: |
./release-${{ steps.platform-abbreviation.outputs.platform }}.sh \
./release-${{ steps.platform-abbreviation.outputs.platform }}.sh ${{ matrix.targets }} \
${{ steps.cargo-flags.outputs.flags }}
working-directory: rust/pact_verifier_cli

- name: Build FFI library
if: |
github.event_name == 'push' ||
github.event_name == 'pull_request' ||
github.event_name == 'workflow_dispatch' ||
startsWith(github.ref, 'refs/tags/libpact_ffi')
shell: bash
run: |
./release-${{ steps.platform-abbreviation.outputs.platform }}.sh \
./release-${{ steps.platform-abbreviation.outputs.platform }}.sh ${{ matrix.targets }} \
${{ steps.cargo-flags.outputs.flags }}
working-directory: rust/pact_ffi

- name: Smoke Test FFI library
if: |
github.event_name == 'push' ||
github.event_name == 'pull_request' ||
github.event_name == 'workflow_dispatch' ||
startsWith(github.ref, 'refs/tags/libpact_ffi')
shell: bash
run: |
./smoke-test.sh ${{ matrix.targets }}
working-directory: ruby

- name: Upload the artifacts
if: |
startsWith(github.ref, 'refs/tags/libpact_ffi') ||
startsWith(github.ref, 'refs/tags/pact_verifier_cli') ||
startsWith(github.ref, 'refs/tags/pact_mock_server_cli')
startsWith(github.ref, 'refs/tags/pact_verifier_cli')
uses: actions/upload-artifact@v4
with:
name: release-artifacts-${{ matrix.operating-system }}
name: release-artifacts-${{ matrix.targets }}
path: rust/release_artifacts
if-no-files-found: error
if-no-files-found: warn

publish:
runs-on: ubuntu-latest
if: |
startsWith(github.ref, 'refs/tags/libpact_ffi') ||
startsWith(github.ref, 'refs/tags/pact_verifier_cli') ||
startsWith(github.ref, 'refs/tags/pact_mock_server_cli')
startsWith(github.ref, 'refs/tags/pact_verifier_cli')
needs: build-release

steps:
Expand Down
89 changes: 89 additions & 0 deletions ruby/detect_os.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
module DetectOS
def self.windows_arm?
return unless !(/cygwin|mswin|mingw|bccwin|wince|emx/ =~ RbConfig::CONFIG['arch']).nil? && !(/arm64/ =~ RbConfig::CONFIG['arch']).nil?
true
end

def self.windows?
return if (/cygwin|mswin|mingw|bccwin|wince|emx/ =~ RbConfig::CONFIG['arch']).nil?
true
end

def self.mac_arm?
return unless !(/darwin/ =~ RbConfig::CONFIG['arch']).nil? && !(/arm64/ =~ RbConfig::CONFIG['arch']).nil?
true
end

def self.mac?
return unless !(/darwin/ =~ RbConfig::CONFIG['arch']).nil? && !(/x86_64/ =~ RbConfig::CONFIG['arch']).nil?
true
end

def self.linux_arm_musl?
return unless !(/linux/ =~ RbConfig::CONFIG['arch']).nil? && !(/aarch64/ =~ RbConfig::CONFIG['arch']).nil? && !(/musl/ =~ RbConfig::CONFIG['arch']).nil?
true
end

def self.linux_musl?
return unless !(/linux/ =~ RbConfig::CONFIG['arch']).nil? && !(/x86_64/ =~ RbConfig::CONFIG['arch']).nil?&& !(/musl/ =~ RbConfig::CONFIG['arch']).nil?
true
end
def self.linux_arm?
return unless !(/linux/ =~ RbConfig::CONFIG['arch']).nil? && !(/aarch64/ =~ RbConfig::CONFIG['arch']).nil?
true
end

def self.linux?
return unless !(/linux/ =~ RbConfig::CONFIG['arch']).nil? && !(/x86_64/ =~ RbConfig::CONFIG['arch']).nil?
true
end

def self.debug?
return if ENV['DEBUG_TARGET'].nil?
true
end

def self.get_bin_path
if debug?
ENV['PACT_FFI_LIBRARY_PATH'].to_s
elsif windows_arm?
File.expand_path("#{__dir__}/../rust/target/aarch64-pc-windows-msvc/release/pact_ffi.dll")
elsif windows?
File.expand_path("#{__dir__}/../rust/target/x86_64-pc-windows-msvc/release/pact_ffi.dll")
elsif mac_arm?
File.expand_path("#{__dir__}/../rust/target/aarch64-apple-darwin/release/libpact_ffi.dylib")
elsif mac?
File.expand_path("#{__dir__}/../rust/target/x86_64-apple-darwin/release/libpact_ffi.dylib")
elsif linux_arm_musl?
File.expand_path("#{__dir__}/../rust/target/aarch64-unknown-linux-musl/release/libpact_ffi.so")
elsif linux_musl?
File.expand_path("#{__dir__}/../rust/target/x86_64-unknown-linux-musl/release/libpact_ffi.so")
elsif linux_arm?
File.expand_path("#{__dir__}/../rust/target/aarch64-unknown-linux-gnu/release/libpact_ffi.so")
elsif linux?
File.expand_path("#{__dir__}/../rust/target/x86_64-unknown-linux-gnu/release/libpact_ffi.so")
else
raise "Detected #{RbConfig::CONFIG['arch']}-- I have no idea what to do with that."
end
end

def self.get_os
if windows_arm?
'win-arm64'
elsif windows?
'win'
elsif mac_arm?
'macos-arm64'
elsif mac?
'linux-x8664'
elsif linux_arm?
'linux-aarch64'
elsif linux?
'linux-x8664'
else
raise "Detected #{RbConfig::CONFIG['arch']}-- I have no idea what to do with that."
end
end
end

ENV['PACT_DEBUG'] ? (puts "Detected platform: #{RbConfig::CONFIG['arch']} \nLoad Path: #{DetectOS.get_bin_path}" ): nil
24 changes: 24 additions & 0 deletions ruby/smoke-test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/bin/bash

set -e
set -x

case $1 in
x86_64-unknown-linux-musl)
docker run --platform=linux/amd64 --rm -v $(pwd)/..:/home -w /home/ruby ruby:alpine ruby test_ffi.rb
;;
aarch64-unknown-linux-musl)
docker run --platform=linux/arm64 --rm -v $(pwd)/..:/home -w /home/ruby ruby:alpine ruby test_ffi.rb
;;
x86_64-unknown-linux-gnu)
docker run --platform=linux/amd64 --rm -v $(pwd)/..:/home -w /home/ruby ruby:slim ruby test_ffi.rb
;;
aarch64-unknown-linux-gnu)
docker run --platform=linux/arm64 --rm -v $(pwd)/..:/home -w /home/ruby ruby:slim ruby test_ffi.rb
;;
aarch64-pc-windows-msvc)
echo unable to test in github actions
exit 0
;;
*) ruby test_ffi.rb ;;
esac
35 changes: 35 additions & 0 deletions ruby/test_ffi.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
require 'fiddle'
require_relative 'detect_os'

lib = Fiddle.dlopen(DetectOS.get_bin_path)

pactffi_version = Fiddle::Function.new(
lib['pactffi_version'],
[],
Fiddle::TYPE_VOIDP
)
pactffi_logger_init = Fiddle::Function.new(
lib['pactffi_logger_init'],
[],
Fiddle::TYPE_VOIDP
)
pactffi_logger_apply = Fiddle::Function.new(
lib['pactffi_logger_apply'],
[],
Fiddle::TYPE_VOIDP
)
pactffi_logger_attach_sink = Fiddle::Function.new(
lib['pactffi_logger_attach_sink'],
[Fiddle::TYPE_VOIDP, Fiddle::TYPE_INT],
Fiddle::TYPE_VOIDP
)
pactffi_log_message = Fiddle::Function.new(
lib['pactffi_log_message'],
[Fiddle::TYPE_VOIDP, Fiddle::TYPE_VOIDP, Fiddle::TYPE_VOIDP],
Fiddle::TYPE_VOIDP
)

pactffi_logger_init.call
pactffi_logger_attach_sink.call('stdout', 5)
pactffi_logger_apply.call
pactffi_log_message.call('pact-ruby-fiddle', 'INFO', "hello from ffi version: #{pactffi_version.call}, platform: #{RUBY_PLATFORM}")
Loading
Loading