Skip to content

Commit 794d346

Browse files
PastaPastaPastaPastaPastaPasta
committed
Merge dashpay#6604: ci: extract "slim" container to run functional tests and add multiprocess and tsan, drop unneeded packages
76f99cd doc: update container documentation (Kittywhiskers Van Gogh) b8b9bb4 ci: split out `multiprocess` linting to a separate step on Actions (Kittywhiskers Van Gogh) 94dedba revert: tentatively drop multiprocess and tsan functional tests (Kittywhiskers Van Gogh) b379708 trivial: `lldbUpdAltArgs` -> `llvmUpdAltArgs` (Kittywhiskers Van Gogh) 90710b3 ci: avoid making copies when collecting logs, misc. script fixes (Kittywhiskers Van Gogh) 89d977d ci: use slim container for functional tests (Kittywhiskers Van Gogh) 22e9acd ci: remove old dependencies, packages not used in building or testing (Kittywhiskers Van Gogh) 630bb0c ci: consolidate package installation in `ci` container (Kittywhiskers Van Gogh) 837d10a ci: extract minimum environment needed to run functional tests (Kittywhiskers Van Gogh) 4bcd093 ci: stop using LLVM setup script to setup repo (Kittywhiskers Van Gogh) a4ce15b ci: rename `ci/Dockerfile`, switch to `devthefuture/dockerfile-x` (Kittywhiskers Van Gogh) 5413955 ci: move packages needed to run `dash-qt` into `develop` container (Kittywhiskers Van Gogh) Pull request description: ## Motivation [dash#6583](dashpay#6583) introduced functional tests to GitHub Actions and due to a combination of limited disk space available on runners and restrictions on ways to free up said disk space (as described in [dash#6583](dashpay#6583)), `multiprocess` and `tsan` builds had to be excluded ([commit](dashpay@3461c14)). Despite measures to reign in disk usage ([source](https://github.com/dashpay/dash/blob/26ea618b81084c74c0a9bb1fed5321012c1ac8ac/ci/dash/slim-workspace.sh)), we are experiencing sporadic failures due to space exhaustion ([build](https://github.com/dashpay/dash/actions/runs/13572578591/job/37941577942), [build](https://github.com/dashpay/dash/actions/runs/13572201698/job/37940403595)). To take care of this, we are going to extract portions of our `ci` container into `ci-slim`, containing only the parts needed to run (sanitizer-enabled) binaries, functional tests and (some) linters. ## Additional Information * We have used [`edrevo/dockerfile-plus`](https://github.com/edrevo/dockerfile-plus) to allow container definitions inherit other container definitions, which has allowed for a ready-to-use interactive container ([`develop`](https://github.com/dashpay/dash/blob/26ea618b81084c74c0a9bb1fed5321012c1ac8ac/contrib/containers/develop/Dockerfile)) to be built on top of our CI container ([`ci`](https://github.com/dashpay/dash/blob/26ea618b81084c74c0a9bb1fed5321012c1ac8ac/contrib/containers/ci/Dockerfile)) with ease (introduced in [dash#4336](dashpay#4336)). Unfortunately, since then, there has been little activity at [`edrevo/dockerfile-plus`](https://github.com/edrevo/dockerfile-plus), last code contribution was ~4 years ago ([commit](edrevo/dockerfile-plus@90dc08d)) and as this PR splits off `ci` to `ci-slim`, it appears `dockerfile-plus` has difficulty in working with `Dockerfiles` that inherit other `Dockerfiles` (see below). <details> <summary>Error message:</summary> ``` $ docker-compose build #0 building with "default" instance using docker driver [...] #1 [container internal] load build definition from Dockerfile #1 transferring dockerfile: 1.57kB done #1 DONE 0.0s [...] #7 [container internal] load build definition from Dockerfile #7 DONE 0.0s failed to solve: failed to create LLB definition: dockerfile parse error line 9: unknown instruction: INCLUDE+ ``` </details> To mitigate this, we are switching over to [`devthefuture/dockerfile-x`](https://codeberg.org/devthefuture/dockerfile-x), which come with minor changes in syntax. * As [`devthefuture/dockerfile-x`](https://codeberg.org/devthefuture/dockerfile-x) (like [`edrevo/dockerfile-plus`](https://github.com/edrevo/dockerfile-plus)) relies on BuildKit, it has been explicitly enabled for GitLab, without which, it fails ([build](https://gitlab.com/dashpay/dash/-/jobs/9276738177#L194)). * As scoping in `Dockerfile`s are relative to the `Dockerfile` itself ([source](https://codeberg.org/devthefuture/dockerfile-x#scoping)), `develop/Dockerfile` can include `ci/Dockerfile` by setting the context one directory behind (which is possible for the root `Dockerfile`) but when `ci/Dockerfile` is parsed, it will search within the `ci` folder and not from the context folder set earlier and so, including `ci-slim/Dockerfile` would come up empty as it would resolve to `ci/ci-slim/Dockerfile`, which doesn't exist. To work around this, both `ci` and `ci-slim` definitions need to share the same directory and to allow for that, `ci/Dockerfile` is now `ci/ci.Dockerfile`, so that `develop/Dockerfile` can set the context a directory behind, find `ci/ci.Dockerfile`, which will search within `ci` for `./ci-slim.Dockerfile`, which will be found successfully. * We have switched away from using the official LLVM installation script to importing the repository manually as we need only a select few packages to run sanitizer-enabled builds and functional tests and the script only gives you a choice between default packages and all packages. * While avoiding LLVM packages entirely in the `ci-slim` container was most desirable, doing so breaks sanitizer builds ([build](https://github.com/kwvg/dash/actions/runs/13585208794/job/37981496193#step:6:2571)) * Switching over to manual import also let us drop a few packages as the script has to probe and work around bugs while our base environment remains relatively similar. * Space occupied by containers. Calculated by modifying `develop/docker-compose.yml` to point at different `Dockerfile`s and building them with `docker compose`, then passing them to [dive](https://github.com/wagoodman/dive) v0.12.0. | Name | Commit | Size | Efficiency Score | Potential Waste | | --------- | -------------------- | ------ | ---------------- | --------------- | | `ci` | `develop` (26ea618) | 4.4 GB | 99% | 74 MB | | `develop` | `develop` (26ea618) | 4.5 GB | 99% | 78 MB | | `ci-slim` | This PR (db6a3fd) | 1.6 GB | 97% | 64 MB | | `ci` | This PR (db6a3fd) | 4.3 GB | 99% | 70 MB | | `develop` | This PR (db6a3fd) | 4.3 GB | 99% | 74 MB | ## Checklist - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] I have added or updated relevant unit/integration/functional/e2e tests **(note: N/A)** - [x] I have made corresponding changes to the documentation - [x] I have assigned this pull request to a milestone _(for repository code-owners and collaborators only)_ ACKs for top commit: UdjinM6: utACK 76f99cd PastaPastaPasta: utACK 76f99cd Tree-SHA512: c0afc570dfc9f6db3dbacbf40b4a2416e993b5e2e503694b9b564a4e00600aa4c290b7c7395f0abc1a7b269c488ed86f4c4c2e64436f38b7ea1653057db83259
2 parents acc2988 + 76f99cd commit 794d346

File tree

15 files changed

+354
-272
lines changed

15 files changed

+354
-272
lines changed

‎.dockerignore‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
**
2-
!contrib/containers/ci/Dockerfile
2+
!contrib/containers/ci/ci.Dockerfile
3+
!contrib/containers/ci/ci-slim.Dockerfile
34
!contrib/containers/deploy/Dockerfile
45
!contrib/containers/develop/Dockerfile

‎.github/workflows/build-container.yml‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,23 @@ name: Build container
22

33
on:
44
workflow_call:
5+
inputs:
6+
context:
7+
description: "Path to use for build context"
8+
required: true
9+
type: string
10+
file:
11+
description: "Path to Dockerfile"
12+
required: true
13+
type: string
14+
name:
15+
description: "Container name"
16+
required: true
17+
type: string
518
outputs:
619
path:
720
description: "Path to built container"
8-
value: ghcr.io/${{ jobs.build.outputs.repo }}/dashcore-ci-runner:${{ jobs.build.outputs.tag }}
21+
value: ghcr.io/${{ jobs.build.outputs.repo }}/${{ inputs.name }}:${{ jobs.build.outputs.tag }}
922

1023
env:
1124
DOCKER_DRIVER: overlay2
@@ -44,15 +57,15 @@ jobs:
4457
- name: Build and push Docker image
4558
uses: docker/build-push-action@v6
4659
with:
47-
context: ./contrib/containers/ci
48-
file: ./contrib/containers/ci/Dockerfile
60+
context: ${{ inputs.context }}
61+
file: ${{ inputs.file }}
4962
push: true
5063
tags: |
51-
ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:${{ hashFiles('./contrib/containers/ci/Dockerfile') }}
52-
ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:${{ steps.prepare.outputs.tag }}
53-
ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:latest
64+
ghcr.io/${{ steps.prepare.outputs.repo }}/${{ inputs.name }}:${{ hashFiles(inputs.file) }}
65+
ghcr.io/${{ steps.prepare.outputs.repo }}/${{ inputs.name }}:${{ steps.prepare.outputs.tag }}
66+
ghcr.io/${{ steps.prepare.outputs.repo }}/${{ inputs.name }}:latest
5467
cache-from: |
55-
type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:${{ hashFiles('./contrib/containers/ci/Dockerfile') }}
56-
type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:${{ steps.prepare.outputs.tag }}
57-
type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo }}/dashcore-ci-runner:latest
68+
type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo }}/${{ inputs.name }}:${{ hashFiles(inputs.file) }}
69+
type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo }}/${{ inputs.name }}:${{ steps.prepare.outputs.tag }}
70+
type=registry,ref=ghcr.io/${{ steps.prepare.outputs.repo }}/${{ inputs.name }}:latest
5871
cache-to: type=inline

‎.github/workflows/build-depends.yml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -70,10 +70,10 @@ jobs:
7070
path: |
7171
depends/built
7272
depends/${{ steps.setup.outputs.HOST }}
73-
key: depends-${{ hashFiles('contrib/containers/ci/Dockerfile') }}-${{ inputs.build-target }}-${{ steps.setup.outputs.DEP_HASH }}-${{ hashFiles('depends/packages/*') }}
73+
key: depends-${{ hashFiles('contrib/containers/ci/ci.Dockerfile') }}-${{ inputs.build-target }}-${{ steps.setup.outputs.DEP_HASH }}-${{ hashFiles('depends/packages/*') }}
7474
restore-keys: |
75-
depends-${{ hashFiles('contrib/containers/ci/Dockerfile') }}-${{ inputs.build-target }}-${{ steps.setup.outputs.DEP_HASH }}-
76-
depends-${{ hashFiles('contrib/containers/ci/Dockerfile') }}-${{ inputs.build-target }}-
75+
depends-${{ hashFiles('contrib/containers/ci/ci.Dockerfile') }}-${{ inputs.build-target }}-${{ steps.setup.outputs.DEP_HASH }}-
76+
depends-${{ hashFiles('contrib/containers/ci/ci.Dockerfile') }}-${{ inputs.build-target }}-
7777
7878
- name: Build depends
7979
run: |

‎.github/workflows/build-src.yml‎

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -70,9 +70,9 @@ jobs:
7070
with:
7171
path: |
7272
/cache
73-
key: ccache-${{ hashFiles('contrib/containers/ci/Dockerfile', 'depends/packages/*') }}-${{ inputs.build-target }}-${{ github.sha }}
73+
key: ccache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'depends/packages/*') }}-${{ inputs.build-target }}-${{ github.sha }}
7474
restore-keys: |
75-
ccache-${{ hashFiles('contrib/containers/ci/Dockerfile', 'depends/packages/*') }}-${{ inputs.build-target }}-
75+
ccache-${{ hashFiles('contrib/containers/ci/ci.Dockerfile', 'depends/packages/*') }}-${{ inputs.build-target }}-
7676
7777
- name: Build source
7878
run: |
@@ -87,6 +87,14 @@ jobs:
8787
ccache -c
8888
shell: bash
8989

90+
- name: Run linters
91+
if: inputs.build-target == 'linux64_multiprocess'
92+
run: |
93+
export BUILD_TARGET="${{ inputs.build-target }}"
94+
source ./ci/dash/matrix.sh
95+
./ci/dash/lint-tidy.sh
96+
shell: bash
97+
9098
- name: Run unit tests
9199
run: |
92100
BASE_OUTDIR="/output"

‎.github/workflows/build.yml‎

Lines changed: 38 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,18 @@ jobs:
1616
container:
1717
name: Build container
1818
uses: ./.github/workflows/build-container.yml
19+
with:
20+
context: ./contrib/containers/ci
21+
file: ./contrib/containers/ci/ci.Dockerfile
22+
name: dashcore-ci-runner
23+
24+
container-slim:
25+
name: Build slim container
26+
uses: ./.github/workflows/build-container.yml
27+
with:
28+
context: ./contrib/containers/ci
29+
file: ./contrib/containers/ci/ci-slim.Dockerfile
30+
name: dashcore-ci-slim
1931

2032
depends-arm-linux:
2133
name: arm-linux-gnueabihf
@@ -158,35 +170,53 @@ jobs:
158170
test-linux64:
159171
name: linux64-test
160172
uses: ./.github/workflows/test-src.yml
161-
needs: [container, src-linux64]
173+
needs: [container-slim, src-linux64]
162174
with:
163175
bundle-key: ${{ needs.src-linux64.outputs.key }}
164176
build-target: linux64
165-
container-path: ${{ needs.container.outputs.path }}
177+
container-path: ${{ needs.container-slim.outputs.path }}
178+
179+
test-linux64_multiprocess:
180+
name: linux64_multiprocess-test
181+
uses: ./.github/workflows/test-src.yml
182+
needs: [container-slim, src-linux64_multiprocess]
183+
with:
184+
bundle-key: ${{ needs.src-linux64_multiprocess.outputs.key }}
185+
build-target: linux64_multiprocess
186+
container-path: ${{ needs.container-slim.outputs.path }}
166187

167188
test-linux64_nowallet:
168189
name: linux64_nowallet-test
169190
uses: ./.github/workflows/test-src.yml
170-
needs: [container, src-linux64_nowallet]
191+
needs: [container-slim, src-linux64_nowallet]
171192
with:
172193
bundle-key: ${{ needs.src-linux64_nowallet.outputs.key }}
173194
build-target: linux64_nowallet
174-
container-path: ${{ needs.container.outputs.path }}
195+
container-path: ${{ needs.container-slim.outputs.path }}
175196

176197
test-linux64_sqlite:
177198
name: linux64_sqlite-test
178199
uses: ./.github/workflows/test-src.yml
179-
needs: [container, src-linux64_sqlite]
200+
needs: [container-slim, src-linux64_sqlite]
180201
with:
181202
bundle-key: ${{ needs.src-linux64_sqlite.outputs.key }}
182203
build-target: linux64_sqlite
183-
container-path: ${{ needs.container.outputs.path }}
204+
container-path: ${{ needs.container-slim.outputs.path }}
205+
206+
test-linux64_tsan:
207+
name: linux64_tsan-test
208+
uses: ./.github/workflows/test-src.yml
209+
needs: [container-slim, src-linux64_tsan]
210+
with:
211+
bundle-key: ${{ needs.src-linux64_tsan.outputs.key }}
212+
build-target: linux64_tsan
213+
container-path: ${{ needs.container-slim.outputs.path }}
184214

185215
test-linux64_ubsan:
186216
name: linux64_ubsan-test
187217
uses: ./.github/workflows/test-src.yml
188-
needs: [container, src-linux64_ubsan]
218+
needs: [container-slim, src-linux64_ubsan]
189219
with:
190220
bundle-key: ${{ needs.src-linux64_ubsan.outputs.key }}
191221
build-target: linux64_ubsan
192-
container-path: ${{ needs.container.outputs.path }}
222+
container-path: ${{ needs.container-slim.outputs.path }}

‎.gitlab-ci.yml‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,9 +29,10 @@ builder-image:
2929
- echo $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
3030
script:
3131
- cd contrib/containers/ci
32+
- export DOCKER_BUILDKIT=1
3233
- docker pull $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG || true
3334
- docker pull $CI_REGISTRY_IMAGE:builder-develop || true
34-
- docker build --cache-from $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG --cache-from $CI_REGISTRY_IMAGE:builder-develop -t $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG -f Dockerfile .
35+
- docker build --cache-from $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG --cache-from $CI_REGISTRY_IMAGE:builder-develop -t $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG -f ci.Dockerfile .
3536
- docker push $CI_REGISTRY_IMAGE:builder-$CI_COMMIT_REF_SLUG
3637

3738
.build-depends-template:
@@ -56,7 +57,7 @@ builder-image:
5657
# Let all branches share the same cache, which is ok because the depends subsystem is able to handle this properly (it works with hashes of all scripts)
5758
key:
5859
files:
59-
- contrib/containers/ci/Dockerfile
60+
- contrib/containers/ci/ci.Dockerfile
6061
prefix: ${CI_JOB_NAME}
6162
paths:
6263
- depends/built
@@ -116,7 +117,7 @@ builder-image:
116117
# Let all branches share the same cache, which is ok because ccache is able to handle it
117118
key:
118119
files:
119-
- contrib/containers/ci/Dockerfile
120+
- contrib/containers/ci/ci.Dockerfile
120121
prefix: ${CI_JOB_NAME}
121122
paths:
122123
- cache/ccache

‎ci/dash/build_src.sh‎

Lines changed: 7 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,7 @@ set -e
1111

1212
source ./ci/dash/matrix.sh
1313

14-
unset CC; unset CXX
15-
unset DISPLAY
14+
unset CC CXX DISPLAY;
1615

1716
if [ "$PULL_REQUEST" != "false" ]; then test/lint/commit-script-check.sh "$COMMIT_RANGE"; fi
1817

@@ -51,6 +50,8 @@ make distdir VERSION="$BUILD_TARGET"
5150
cd "dashcore-$BUILD_TARGET"
5251
bash -c "./configure $BITCOIN_CONFIG_ALL $BITCOIN_CONFIG" || ( cat config.log && false)
5352

53+
# This step influences compilation and therefore will always be a part of the
54+
# compile step
5455
if [ "${RUN_TIDY}" = "true" ]; then
5556
MAYBE_BEAR="bear --config src/.bear-tidy-config"
5657
MAYBE_TOKEN="--"
@@ -65,22 +66,10 @@ if [ -n "$USE_VALGRIND" ]; then
6566
"${BASE_ROOT_DIR}/ci/test/wrap-valgrind.sh"
6667
fi
6768

68-
if [ "${RUN_TIDY}" = "true" ]; then
69-
set -eo pipefail
70-
cd src
71-
( run-clang-tidy -quiet "${MAKEJOBS}" ) | grep -C5 "error"
72-
cd ..
73-
iwyu_tool.py \
74-
"src/compat" \
75-
"src/init" \
76-
"src/rpc/fees.cpp" \
77-
"src/rpc/signmessage.cpp" \
78-
-p . "${MAKEJOBS}" \
79-
-- -Xiwyu --cxx17ns -Xiwyu --mapping_file="${BASE_ROOT_DIR}/contrib/devtools/iwyu/bitcoin.core.imp" \
80-
|& tee "/tmp/iwyu_ci.out"
81-
cd src
82-
fix_includes.py --nosafe_headers < /tmp/iwyu_ci.out
83-
git --no-pager diff
69+
# GitHub Actions can segment a job into steps, linting is a separate step
70+
# so Actions runners will perform this step separately.
71+
if [ "${RUN_TIDY}" = "true" ] && [ "${GITHUB_ACTIONS}" != "true" ]; then
72+
"${BASE_ROOT_DIR}/ci/dash/lint-tidy.sh"
8473
fi
8574

8675
if [ "$RUN_SECURITY_TESTS" = "true" ]; then

‎ci/dash/lint-tidy.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/usr/bin/env bash
2+
# Copyright (c) 2025 The Dash Core developers
3+
# Distributed under the MIT software license, see the accompanying
4+
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
5+
6+
export LC_ALL=C.UTF-8
7+
8+
set -eo pipefail
9+
10+
# Warning: This script does not generate the compilation database these linters rely
11+
# only on nor do they set the requisite build parameters. Make sure you do
12+
# that *before* running this script.
13+
14+
cd "${BASE_ROOT_DIR}/build-ci/dashcore-${BUILD_TARGET}/src"
15+
( run-clang-tidy -quiet "${MAKEJOBS}" ) | grep -C5 "error"
16+
17+
cd "${BASE_ROOT_DIR}/build-ci/dashcore-${BUILD_TARGET}"
18+
iwyu_tool.py \
19+
"src/compat" \
20+
"src/init" \
21+
"src/rpc/fees.cpp" \
22+
"src/rpc/signmessage.cpp" \
23+
-p . "${MAKEJOBS}" \
24+
-- -Xiwyu --cxx17ns -Xiwyu --mapping_file="${BASE_ROOT_DIR}/contrib/devtools/iwyu/bitcoin.core.imp" \
25+
2>&1 | tee "/tmp/iwyu_ci.out"
26+
27+
cd "${BASE_ROOT_DIR}/build-ci/dashcore-${BUILD_TARGET}/src"
28+
fix_includes.py --nosafe_headers < /tmp/iwyu_ci.out
29+
git --no-pager diff

‎ci/dash/slim-workspace.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ TARGETS=(
3232
"build-ci/dashcore-${BUILD_TARGET}/src/qt/qrc_dash_locale.cpp"
3333
)
3434

35-
# Delete directories we don't need
35+
# Delete what we don't need
3636
for target in "${TARGETS[@]}"
3737
do
3838
if [[ -d "${target}" ]] || [[ -f "${target}" ]]; then

‎ci/dash/test_integrationtests.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ if [ "$BASEDIR" != "" ]; then
6666
for f in $LOGFILES; do
6767
d2="testlogs/$d/$(dirname "$f")"
6868
mkdir -p "$d2"
69-
cp "testdatadirs/$BASEDIR/$d/$f" "$d2/"
69+
mv "testdatadirs/$BASEDIR/$d/$f" "$d2/"
7070
done
7171
done
7272
fi

0 commit comments

Comments
 (0)