Skip to content

CI: some improvements for faster PR builds #6397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 34 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
c21a979
use quiet instead of verbose (revert #3634)
cce Jul 30, 2025
90ad537
use custom setup-go action
cce Jul 30, 2025
1412715
libsodium caching for reviewdog and tools
cce Jul 30, 2025
68a4a2f
Run unit tests right away without waiting for build
cce Jul 30, 2025
2dee5d0
Don't use gimme
cce Jul 30, 2025
f11f8d2
remove legacy GOCACHE NFS workaround (from #1072)
cce Jul 30, 2025
8ccb29d
use new libsodium make target in before_build.sh
cce Jul 30, 2025
65977f0
remove unnecessary build job and workspace archiving
cce Jul 30, 2025
6f2e54d
skip apt install unless we need expect
cce Jul 31, 2025
08eb186
skip building local packages for e2e
cce Jul 31, 2025
5c83727
skip double e2e binary builds, skip apt update for expect
cce Jul 31, 2025
864d8fb
skip man-db rebuild
cce Jul 31, 2025
99ca29d
bump e2e_expect to 9 shards
cce Jul 31, 2025
79d7fb6
build-e2e in parallel
cce Jul 31, 2025
8e530fe
rename build.yml
cce Jul 31, 2025
2b94b9e
shard e2e_subs
cce Jul 31, 2025
22002bd
also build regular algod and goal so e2e test can run goal helptest s…
cce Aug 1, 2025
ef4c04b
move CI env var checks and package management into custom action
cce Aug 1, 2025
c1ffa29
verify for e2e_subs sharding
cce Aug 1, 2025
f474bf9
add reusable slack action
cce Aug 1, 2025
0002a7f
bump sharding a little
cce Aug 1, 2025
9bcffcf
use slack notification action in ci-nightly
cce Aug 1, 2025
f609a9f
update codegen_verification to use setup-go and setup-test
cce Aug 1, 2025
c44e435
fix codegen_verification job
cce Aug 1, 2025
99deaa9
add summary job
cce Aug 1, 2025
25bdf52
ensure failed tests make it into summary
cce Aug 1, 2025
a0e700a
use "make libsodium" in tools and reviewdog workflows
cce Aug 2, 2025
08d7b3b
manual mod and build caching
cce Aug 2, 2025
fce6f5a
make cache keys arch specific too
cce Aug 2, 2025
f39f32f
bump rebuild to check cache performance
cce Aug 2, 2025
fd45874
use cache just for ci-pr.yml
cce Aug 2, 2025
da119cd
bump for rebuild to test cache
cce Aug 2, 2025
d4904fe
trigger another cache rebuild a day later to test key prefix matching
cce Aug 3, 2025
82a45c9
fix report summary to not list null
cce Aug 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions .github/actions/setup-go/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
name: 'Setup Go with caching'
description: 'Set up Go with mod and build caching'
inputs:
cache-prefix:
description: 'Cache prefix for advanced caching (e.g., "test", "build-e2e", "buildsrc")'
required: false
default: ''
runs:
using: 'composite'
steps:
- name: Get Go version
id: go_version
run: echo "GO_VERSION=$(./scripts/get_golang_version.sh)" >> $GITHUB_ENV
shell: bash
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
cache: ${{ inputs.cache-prefix == '' }}
# branch-aware caching with fallback key prefixes following examples from:
# https://github.com/algorand/go-algorand/blob/ab03c94d2008dd349b761ef2b949e17b361996a3/.circleci/config.yml#L407-L439
# https://danp.net/posts/github-actions-go-cache/
- name: Set cache date and paths
if: inputs.cache-prefix != ''
shell: bash
run: |
echo "GO_CACHE_DATE=$(date +'%Y-%m-%d')" >> $GITHUB_ENV
echo "GOMODCACHE=$(go env GOMODCACHE)" >> $GITHUB_ENV
echo "GOCACHE=$(go env GOCACHE)" >> $GITHUB_ENV
- name: Cache Go modules
if: inputs.cache-prefix != ''
uses: actions/cache@v4
with:
path: ${{ env.GOMODCACHE }}
key: ${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-mod-${{ github.ref_name }}-${{ hashFiles('**/go.sum') }}-${{ env.GO_CACHE_DATE }}
restore-keys: |
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-mod-${{ github.ref_name }}-${{ hashFiles('**/go.sum') }}-
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-mod-${{ github.ref_name }}-
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-mod-master-
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-mod-
- name: Cache Go build cache
if: inputs.cache-prefix != ''
uses: actions/cache@v4
with:
path: ${{ env.GOCACHE }}
key: ${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-build-${{ github.ref_name }}-${{ hashFiles('**/go.sum') }}-${{ env.GO_CACHE_DATE }}
restore-keys: |
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-build-${{ github.ref_name }}-${{ hashFiles('**/go.sum') }}-
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-build-${{ github.ref_name }}-
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-build-master-
${{ inputs.cache-prefix }}-${{ runner.os }}-${{ runner.arch }}-go-build-
- name: Download Go modules
if: inputs.cache-prefix != ''
shell: bash
run: go mod download
40 changes: 40 additions & 0 deletions .github/actions/setup-test/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: 'Set up Test Environment'
description: 'Set up testing environment for all platforms'
inputs:
install-expect:
description: 'Whether to install expect package'
required: false
default: 'false'
runs:
using: 'composite'
steps:
- name: Cache libsodium
uses: actions/cache@v4
with:
path: crypto/libs
key: libsodium-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('crypto/libsodium-fork/**') }}
- name: Build libsodium
run: make libsodium
shell: bash
- name: Configure development environment
if: runner.os != 'Linux'
run: ./scripts/configure_dev.sh
shell: bash
# GHA Linux VMs provide all needed apt packages, except expect
- name: Install expect package
if: inputs.install-expect == 'true' && runner.os == 'Linux'
run: |
echo 'set man-db/auto-update false' | sudo debconf-communicate >/dev/null
sudo dpkg-reconfigure -f noninteractive man-db
sudo apt-get install -y --no-upgrade expect
shell: bash
- name: Cache gotestsum
id: cache-gotestsum
uses: actions/cache@v4
with:
path: ~/go/bin/gotestsum
key: gotestsum-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('scripts/buildtools/versions') }}
- name: Install gotestsum
if: steps.cache-gotestsum.outputs.cache-hit != 'true'
run: ./scripts/buildtools/install_buildtools.sh -o "gotest.tools/gotestsum"
shell: bash
35 changes: 35 additions & 0 deletions .github/actions/slack-notify/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: 'Slack Failure Notification'
description: 'Send failure notification to Slack'
inputs:
job-type:
description: 'Type of job that failed (e.g., Test, Integration Test, etc.)'
required: true
build-type:
description: 'Type of build (e.g., PR Build, Nightly Build)'
required: true
details:
description: 'Additional job-specific details (formatted string)'
required: false
default: ''
runs:
using: 'composite'
steps:
- name: Send Slack notification
if: env.SLACK_WEBHOOK != ''
uses: slackapi/slack-github-action@v2.1.0
with:
webhook: ${{ env.SLACK_WEBHOOK }}
webhook-type: webhook-trigger
payload: |
{
"text": "🚨 ${{ inputs.job-type }} Failure Alert",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*${{ inputs.job-type }} Failure in ${{ inputs.build-type }}*\n\n• Job Type: `${{ github.job }}`\n• Platform: `${{ matrix.platform }}`${{ inputs.details != '' && format('\n{0}', inputs.details) || '' }}\n• Run URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}"
}
}
]
}
9 changes: 5 additions & 4 deletions .github/workflows/benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,12 +18,13 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version-file: 'go.mod'
- name: Set up Go
uses: ./.github/actions/setup-go
- run: go version
- name: Build go-algorand
run: scripts/travis/build.sh
run: |
export SKIP_GO_INSTALLATION=True
scripts/travis/build.sh
# BenchmarkUintMath - Serves as a proxy for AVM `eval` performance.
# Performance degradations suggest either or both: (1) op code
# degradation, (2) `eval` degradation. (2) suggests a broader performance
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,19 +20,14 @@ jobs:
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Determine Go version
id: go_version
run: echo "GO_VERSION=$(./scripts/get_golang_version.sh)" >> $GITHUB_ENV
- name: Install golang
uses: actions/setup-go@v5
with:
go-version: ${{ env.GO_VERSION }}
- name: Set up Go
uses: ./.github/actions/setup-go
- name: Restore libsodium from cache
id: cache-libsodium
uses: actions/cache@v4
with:
path: crypto/libs
key: libsodium-fork-v2-${{ runner.os }}-${{ hashFiles('crypto/libsodium-fork/**') }}
key: libsodium-fork-v2-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('crypto/libsodium-fork/**') }}
- name: Build
run: |
export ALGORAND_DEADLOCK=enable
Expand Down
Loading
Loading