-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: Migrate CI scripts to CMake #142
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,254 @@ | ||
# Copyright (c) 2023 The Bitcoin Core developers | ||
# Distributed under the MIT software license, see the accompanying | ||
# file COPYING or http://www.opensource.org/licenses/mit-license.php. | ||
|
||
name: CI | ||
on: | ||
# See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#pull_request. | ||
pull_request: | ||
# See: https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#push. | ||
push: | ||
branches: | ||
- '**' | ||
tags-ignore: | ||
- '**' | ||
|
||
concurrency: | ||
group: ${{ github.event_name != 'pull_request' && github.run_id || github.ref }} | ||
cancel-in-progress: true | ||
|
||
env: | ||
DANGER_RUN_CI_ON_HOST: 1 | ||
CI_FAILFAST_TEST_LEAVE_DANGLING: 1 # GHA does not care about dangling processes and setting this variable avoids killing the CI script itself on error | ||
MAKEJOBS: '-j10' | ||
|
||
jobs: | ||
test-each-commit: | ||
name: 'test each commit' | ||
runs-on: ubuntu-22.04 | ||
if: github.event_name == 'pull_request' && github.event.pull_request.commits != 1 | ||
timeout-minutes: 360 # Use maximum time, see https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idtimeout-minutes. Assuming a worst case time of 1 hour per commit, this leads to a --max-count=6 below. | ||
env: | ||
MAX_COUNT: 6 | ||
steps: | ||
- name: Determine fetch depth | ||
run: echo "FETCH_DEPTH=$((${{ github.event.pull_request.commits }} + 2))" >> "$GITHUB_ENV" | ||
- uses: actions/checkout@v4 | ||
with: | ||
ref: ${{ github.event.pull_request.head.sha }} | ||
fetch-depth: ${{ env.FETCH_DEPTH }} | ||
- name: Determine commit range | ||
run: | | ||
# Checkout HEAD~ and find the test base commit | ||
# Checkout HEAD~ because it would be wasteful to rerun tests on the PR | ||
# head commit that are already run by other jobs. | ||
git checkout HEAD~ | ||
# Figure out test base commit by listing ancestors of HEAD, excluding | ||
# ancestors of the most recent merge commit, limiting the list to the | ||
# newest MAX_COUNT ancestors, ordering it from oldest to newest, and | ||
# taking the first one. | ||
# | ||
# If the branch contains up to MAX_COUNT ancestor commits after the | ||
# most recent merge commit, all of those commits will be tested. If it | ||
# contains more, only the most recent MAX_COUNT commits will be | ||
# tested. | ||
# | ||
# In the command below, the ^@ suffix is used to refer to all parents | ||
# of the merge commit as described in: | ||
# https://git-scm.com/docs/git-rev-parse#_other_rev_parent_shorthand_notations | ||
# and the ^ prefix is used to exclude these parents and all their | ||
# ancestors from the rev-list output as described in: | ||
# https://git-scm.com/docs/git-rev-list | ||
echo "TEST_BASE=$(git rev-list -n$((${{ env.MAX_COUNT }} + 1)) --reverse HEAD ^$(git rev-list -n1 --merges HEAD)^@ | head -1)" >> "$GITHUB_ENV" | ||
- run: | | ||
sudo apt-get update | ||
sudo apt-get install clang-15 ccache build-essential libtool autotools-dev automake pkg-config bsdmainutils python3-zmq libevent-dev libboost-dev libsqlite3-dev libdb++-dev systemtap-sdt-dev libminiupnpc-dev libnatpmp-dev qtbase5-dev qttools5-dev qttools5-dev-tools qtwayland5 libqrencode-dev -y | ||
- name: Compile and run tests | ||
run: | | ||
# Run tests on commits after the last merge commit and before the PR head commit | ||
# Use clang++, because it is a bit faster and uses less memory than g++ | ||
git rebase --exec "echo Running test-one-commit on \$( git log -1 ) && ./autogen.sh && CC=clang-15 CXX=clang++-15 ./configure && make clean && make -j $(nproc) check && ./test/functional/test_runner.py -j $(( $(nproc) * 2 ))" ${{ env.TEST_BASE }} | ||
|
||
macos-native-x86_64: | ||
name: 'macOS 13 native, x86_64, no depends, sqlite only, gui' | ||
# Use latest image, but hardcode version to avoid silent upgrades (and breaks). | ||
# See: https://github.com/actions/runner-images#available-images. | ||
runs-on: macos-13 | ||
|
||
# No need to run on the read-only mirror, unless it is a PR. | ||
if: github.repository != 'bitcoin-core/gui' || github.event_name == 'pull_request' | ||
|
||
timeout-minutes: 120 | ||
|
||
env: | ||
FILE_ENV: './ci/test/00_setup_env_mac_native.sh' | ||
BASE_ROOT_DIR: ${{ github.workspace }} | ||
|
||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
|
||
- name: Clang version | ||
run: | | ||
sudo xcode-select --switch /Applications/Xcode_15.0.app | ||
clang --version | ||
|
||
- name: Install Homebrew packages | ||
env: | ||
HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: 1 | ||
run: | | ||
# A workaround for "The `brew link` step did not complete successfully" error. | ||
brew install python@3 || brew link --overwrite python@3 | ||
brew install automake libtool pkg-config gnu-getopt ccache boost libevent miniupnpc libnatpmp zeromq qt@5 qrencode | ||
|
||
- name: Set Ccache directory | ||
run: echo "CCACHE_DIR=${RUNNER_TEMP}/ccache_dir" >> "$GITHUB_ENV" | ||
|
||
- name: Restore Ccache cache | ||
id: ccache-cache | ||
uses: actions/cache/restore@v4 | ||
with: | ||
path: ${{ env.CCACHE_DIR }} | ||
key: ${{ github.job }}-ccache-${{ github.run_id }} | ||
restore-keys: ${{ github.job }}-ccache- | ||
|
||
- name: CI script | ||
run: ./ci/test_run_all.sh | ||
|
||
- name: Save Ccache cache | ||
uses: actions/cache/save@v4 | ||
if: github.event_name != 'pull_request' && steps.ccache-cache.outputs.cache-hit != 'true' | ||
with: | ||
path: ${{ env.CCACHE_DIR }} | ||
# https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache | ||
key: ${{ github.job }}-ccache-${{ github.run_id }} | ||
|
||
win64-native: | ||
name: 'Win64 native, VS 2022' | ||
# Use latest image, but hardcode version to avoid silent upgrades (and breaks). | ||
# See: https://github.com/actions/runner-images#available-images. | ||
runs-on: windows-2022 | ||
|
||
# No need to run on the read-only mirror, unless it is a PR. | ||
if: github.repository != 'bitcoin-core/gui' || github.event_name == 'pull_request' | ||
|
||
env: | ||
CCACHE_MAXSIZE: '200M' | ||
PYTHONUTF8: 1 | ||
TEST_RUNNER_TIMEOUT_FACTOR: 40 | ||
|
||
steps: | ||
- name: Checkout | ||
uses: actions/checkout@v4 | ||
|
||
- name: Remove non-MSVC tool installations | ||
run: | | ||
Remove-Item -Path "$env:ProgramFiles/CMake" -Recurse -Force | ||
|
||
- name: Configure Developer Command Prompt for Microsoft Visual C++ | ||
# Using microsoft/setup-msbuild is not enough. | ||
uses: ilammy/msvc-dev-cmd@v1 | ||
with: | ||
arch: x64 | ||
|
||
- name: Get tool information | ||
run: | | ||
cmake --version | Out-File -FilePath "cmake_version" | ||
Get-Content -Path "cmake_version" | ||
Write-Output "---" | ||
msbuild -version | Out-File -FilePath "$env:GITHUB_WORKSPACE\msbuild_version" | ||
Get-Content -Path "$env:GITHUB_WORKSPACE\msbuild_version" | ||
$env:VCToolsVersion | Out-File -FilePath "$env:GITHUB_WORKSPACE\toolset_version" | ||
Write-Host "VCToolsVersion $(Get-Content -Path "$env:GITHUB_WORKSPACE\toolset_version")" | ||
py -3 --version | ||
Write-Host "PowerShell version $($PSVersionTable.PSVersion.ToString())" | ||
|
||
- name: Install Ccache | ||
run: | | ||
choco install --yes --no-progress ccache | ||
|
||
- name: Using vcpkg with MSBuild | ||
run: | | ||
Set-Location "$env:VCPKG_INSTALLATION_ROOT" | ||
Add-Content -Path "triplets\x64-windows.cmake" -Value "set(VCPKG_BUILD_TYPE release)" | ||
Add-Content -Path "triplets\x64-windows.cmake" -Value "set(VCPKG_PLATFORM_TOOLSET_VERSION $env:VCToolsVersion)" | ||
|
||
- name: vcpkg tools cache | ||
uses: actions/cache@v4 | ||
with: | ||
path: C:/vcpkg/downloads/tools | ||
key: ${{ github.job }}-vcpkg-tools | ||
|
||
- name: Restore vcpkg binary cache | ||
uses: actions/cache/restore@v4 | ||
id: vcpkg-binary-cache | ||
with: | ||
path: ~/AppData/Local/vcpkg/archives | ||
key: ${{ github.job }}-vcpkg-binary-${{ hashFiles('cmake_version', 'msbuild_version', 'toolset_version', 'vcpkg.json') }} | ||
|
||
- name: Generate build system | ||
run: | | ||
cmake -B build --preset vs2022 -DCMAKE_TOOLCHAIN_FILE="$env:VCPKG_INSTALLATION_ROOT\scripts\buildsystems\vcpkg.cmake" -DWERROR=ON | ||
|
||
- name: Save vcpkg binary cache | ||
uses: actions/cache/save@v4 | ||
if: github.event_name != 'pull_request' && steps.vcpkg-binary-cache.outputs.cache-hit != 'true' | ||
with: | ||
path: ~/AppData/Local/vcpkg/archives | ||
key: ${{ github.job }}-vcpkg-binary-${{ hashFiles('cmake_version', 'msbuild_version', 'toolset_version', 'vcpkg.json') }} | ||
|
||
- name: Restore Ccache cache | ||
id: ccache-cache | ||
uses: actions/cache/restore@v4 | ||
with: | ||
path: ~/AppData/Local/ccache | ||
key: ${{ github.job }}-ccache- | ||
|
||
- name: Build | ||
working-directory: build | ||
run: | | ||
ccache --zero-stats | ||
cmake --build . -j $env:NUMBER_OF_PROCESSORS --config Release | ||
|
||
- name: Ccache stats | ||
run: | | ||
ccache --version | head -n 1 | ||
ccache --show-stats --verbose | ||
|
||
- name: Save Ccache cache | ||
uses: actions/cache/save@v4 | ||
if: github.event_name != 'pull_request' && steps.ccache-cache.outputs.cache-hit != 'true' | ||
with: | ||
path: ~/AppData/Local/ccache | ||
# https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache | ||
key: ${{ github.job }}-ccache-${{ github.run_id }} | ||
|
||
- name: Run test suite | ||
working-directory: build | ||
run: | | ||
ctest -j $env:NUMBER_OF_PROCESSORS -C Release | ||
|
||
- name: Run functional tests | ||
working-directory: build | ||
env: | ||
BITCOIND: '${{ github.workspace }}\build\src\Release\bitcoind.exe' | ||
BITCOINCLI: '${{ github.workspace }}\build\src\Release\bitcoin-cli.exe' | ||
BITCOINUTIL: '${{ github.workspace }}\build\src\Release\bitcoin-util.exe' | ||
BITCOINWALLET: '${{ github.workspace }}\build\src\Release\bitcoin-wallet.exe' | ||
TEST_RUNNER_EXTRA: ${{ github.event_name != 'pull_request' && '--extended' || '' }} | ||
shell: cmd | ||
run: py -3 test\functional\test_runner.py --jobs %NUMBER_OF_PROCESSORS% --ci --quiet --tmpdirprefix=%RUNNER_TEMP% --combinedlogslen=99999999 --timeout-factor=%TEST_RUNNER_TIMEOUT_FACTOR% %TEST_RUNNER_EXTRA% | ||
|
||
- name: Clone fuzz corpus | ||
run: | | ||
git clone --depth=1 https://github.com/bitcoin-core/qa-assets "$env:RUNNER_TEMP\qa-assets" | ||
Set-Location "$env:RUNNER_TEMP\qa-assets" | ||
Write-Host "Using qa-assets repo from commit ..." | ||
git log -1 | ||
|
||
- name: Run fuzz binaries | ||
working-directory: build | ||
env: | ||
BITCOINFUZZ: '${{ github.workspace }}\build\src\test\fuzz\Release\fuzz.exe' | ||
shell: cmd | ||
run: py -3 test\fuzz\test_runner.py --par %NUMBER_OF_PROCESSORS% --loglevel DEBUG %RUNNER_TEMP%\qa-assets\fuzz_seed_corpus |
Original file line number | Diff line number | Diff line change | ||
---|---|---|---|---|
|
@@ -17,10 +17,11 @@ else | |||
fi | ||||
|
||||
export CONTAINER_NAME=ci_native_asan | ||||
export PACKAGES="systemtap-sdt-dev clang-18 llvm-18 libclang-rt-18-dev python3-zmq qtbase5-dev qttools5-dev-tools libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libqrencode-dev libsqlite3-dev ${BPFCC_PACKAGE}" | ||||
export PACKAGES="systemtap-sdt-dev clang-18 llvm-18 libclang-rt-18-dev python3-zmq qtbase5-dev qttools5-dev qttools5-dev-tools libevent-dev libboost-dev libdb5.3++-dev libminiupnpc-dev libnatpmp-dev libzmq3-dev libqrencode-dev libsqlite3-dev ${BPFCC_PACKAGE}" | ||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looks like this redundant? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It follows the build docs: Line 84 in ecd2365
Otherwise, CMakes fails to find There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Right, please resolve. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I guess the change could also be submitted upstream standalone, maybe along with similar changes that are unrelated to cmake? But either way seems fine. |
||||
export NO_DEPENDS=1 | ||||
export GOAL="install" | ||||
export BITCOIN_CONFIG="--enable-usdt --enable-zmq --with-incompatible-bdb --with-gui=qt5 \ | ||||
CPPFLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER' \ | ||||
--with-sanitizers=address,float-divide-by-zero,integer,undefined \ | ||||
CC='clang-18 -ftrivial-auto-var-init=pattern' CXX='clang++-18 -ftrivial-auto-var-init=pattern'" | ||||
export BITCOIN_CONFIG="-DWITH_USDT=ON -DWITH_ZMQ=ON -DWARN_INCOMPATIBLE_BDB=OFF -DBUILD_GUI=ON \ | ||||
-DSANITIZERS=address,float-divide-by-zero,integer,undefined \ | ||||
-DCMAKE_C_COMPILER=clang-18 -DCMAKE_CXX_COMPILER=clang++-18 \ | ||||
-DCMAKE_C_FLAGS='-ftrivial-auto-var-init=pattern' \ | ||||
-DCMAKE_CXX_FLAGS='-DARENA_DEBUG -DDEBUG_LOCKORDER -ftrivial-auto-var-init=pattern'" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these not going into APPEND flags because we don't add
-Wdocumentation
ourselves? Meaning that the primary use-case for APPEND_CXXFLAGS is specifically when we:?
If so, it would be nice to document exactly that.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait, we do add
-Wdocumentation
though. So why doesn't this have to be in APPEND_CXXFLAGS?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We have to use
APPEND_CXXFLAGS
for-Wno-documentation
.-Wno-error=documentation
works withCMAKE_CXX_FLAGS
just fine.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, ok. I would've thought that order matters there. But that makes sense.