Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GHW for building and publishing docker images #1391

Merged
merged 29 commits into from
Sep 6, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
d2702c0
add ghw and scripts for docker image deployment
EgorPopelyaev Sep 1, 2023
c3b0222
debug
EgorPopelyaev Sep 4, 2023
01f543c
add permissions for content
EgorPopelyaev Sep 4, 2023
365c744
fix path to the bin folder
EgorPopelyaev Sep 4, 2023
2870fd4
add tags
EgorPopelyaev Sep 4, 2023
c42eace
rename env
EgorPopelyaev Sep 4, 2023
b0dfb3e
fix path to docker file
EgorPopelyaev Sep 4, 2023
b71c3cf
make polkadot-parachain executable
EgorPopelyaev Sep 4, 2023
6c3c726
fix typo
EgorPopelyaev Sep 4, 2023
40c16d9
fix more typos
EgorPopelyaev Sep 4, 2023
aecb1cf
test
EgorPopelyaev Sep 4, 2023
fe9eb58
revert back use of working directory
EgorPopelyaev Sep 4, 2023
6ebf9d0
mke bin executable in the artifacts folder
EgorPopelyaev Sep 4, 2023
22f0a73
use cd instead of working directory
EgorPopelyaev Sep 4, 2023
bae25f4
change path to cash
EgorPopelyaev Sep 4, 2023
b933aef
fix path to cash
EgorPopelyaev Sep 4, 2023
554caba
change cache key
EgorPopelyaev Sep 4, 2023
f32e2b7
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev Sep 4, 2023
c8a94d4
delete old flows
EgorPopelyaev Sep 4, 2023
635252b
addressed PR comments
EgorPopelyaev Sep 5, 2023
ebc7f06
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev Sep 5, 2023
09761f6
fix path
EgorPopelyaev Sep 5, 2023
97de854
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev Sep 5, 2023
eb7753b
reorg docker files
EgorPopelyaev Sep 6, 2023
8748d67
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev Sep 6, 2023
7a2c19b
cleanup
EgorPopelyaev Sep 6, 2023
56ad8eb
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev Sep 6, 2023
a00287c
reog dockerfiles
EgorPopelyaev Sep 6, 2023
aa37acc
Merge branch 'master' of https://github.com/paritytech/polkadot-sdk i…
EgorPopelyaev Sep 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
reog dockerfiles
  • Loading branch information
EgorPopelyaev committed Sep 6, 2023
commit a00287c72114f990b86a66bcaeddc6359f1dbdab
2 changes: 1 addition & 1 deletion .github/workflows/release-50_publish-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -176,7 +176,7 @@ jobs:
ARTIFACTS_FOLDER: ./release-artifacts
IMAGE_NAME: ${{ env.BINARY }}
OWNER: ${{ env.DOCKER_OWNER }}
DOCKERFILE: docker/polkadot-parachain/polkadot-parachain_injected.Dockerfile
DOCKERFILE: docker/dockerfiles/polkadot-parachain/polkadot-parachain_injected.Dockerfile
TAGS: ${{ join(steps.fetch_rc_refs.outputs.*, ',') || join(steps.fetch_release_refs.outputs.*, ',') }}
run: |
ls -al
Expand Down
2 changes: 1 addition & 1 deletion .gitlab/pipeline/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -328,7 +328,7 @@ build-linux-substrate:
cut -d ' ' -f 2 | tee ./artifacts/substrate/VERSION;
fi
- sha256sum ./artifacts/substrate/substrate | tee ./artifacts/substrate/substrate.sha256
- cp -r ./docker/substrate_injected.Dockerfile ./artifacts/substrate/
- cp -r ./docker/dockerfiles/substrate_injected.Dockerfile ./artifacts/substrate/
# - printf '\n# building node-template\n\n'
# - ./scripts/ci/node-template-release.sh ./artifacts/substrate/substrate-node-template.tar.gz

Expand Down
18 changes: 9 additions & 9 deletions .gitlab/pipeline/publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ build-push-image-polkadot-parachain-debug:
- job: build-linux-stable-cumulus
artifacts: true
variables:
DOCKERFILE: "docker/polkadot-parachain/polkadot-parachain-debug_unsigned_injected.Dockerfile"
DOCKERFILE: "docker/dockerfiles/polkadot-parachain/polkadot-parachain-debug_unsigned_injected.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/polkadot-parachain-debug"

build-push-image-test-parachain:
Expand All @@ -48,7 +48,7 @@ build-push-image-test-parachain:
- job: build-test-parachain
artifacts: true
variables:
DOCKERFILE: "docker/test-parachain_injected.Dockerfile"
DOCKERFILE: "docker/dockerfiles/test-parachain_injected.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/test-parachain"
# publish-s3:
# stage: publish
Expand Down Expand Up @@ -114,7 +114,7 @@ build-push-image-polkadot-debug:
- job: build-linux-stable
artifacts: true
variables:
DOCKERFILE: "docker/polkadot/polkadot_injected_debug.Dockerfile"
DOCKERFILE: "docker/dockerfiles/polkadot/polkadot_injected_debug.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/polkadot-debug"

build-push-image-colander:
Expand All @@ -127,7 +127,7 @@ build-push-image-colander:
- job: build-test-collators
artifacts: true
variables:
DOCKERFILE: "docker/collator_injected.Dockerfile"
DOCKERFILE: "docker/dockerfiles/collator_injected.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/colander"

build-push-image-malus:
Expand All @@ -140,7 +140,7 @@ build-push-image-malus:
- job: build-malus
artifacts: true
variables:
DOCKERFILE: "docker/malus_injected.Dockerfile"
DOCKERFILE: "docker/dockerfiles/malus_injected.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/malus"

build-push-image-substrate-pr:
Expand All @@ -153,7 +153,7 @@ build-push-image-substrate-pr:
- job: build-linux-substrate
artifacts: true
variables:
DOCKERFILE: "docker/substrate_injected.Dockerfile"
DOCKERFILE: "docker/dockerfiles/substrate_injected.Dockerfile"
IMAGE_NAME: "docker.io/paritypr/substrate"
# old way

Expand Down Expand Up @@ -201,7 +201,7 @@ build-push-image-substrate-pr:
# GIT_STRATEGY: none
# DOCKER_USER: ${PARITYPR_USER}
# DOCKER_PASS: ${PARITYPR_PASS}
# # docker/polkadot/polkadot_injected_debug.Dockerfile
# # docker/dockerfiles/polkadot/polkadot_injected_debug.Dockerfile
# DOCKERFILE: polkadot_injected_debug.Dockerfile
# IMAGE_NAME: docker.io/paritypr/polkadot-debug
# needs:
Expand Down Expand Up @@ -230,7 +230,7 @@ build-push-image-substrate-pr:
# GIT_STRATEGY: none
# DOCKER_USER: ${PARITYPR_USER}
# DOCKER_PASS: ${PARITYPR_PASS}
# # scripts/ci/dockerfiles/collator_injected.Dockerfile
# # docker/dockerfiles/collator_injected.Dockerfile
# DOCKERFILE: collator_injected.Dockerfile
# IMAGE_NAME: docker.io/paritypr/colander
# needs:
Expand Down Expand Up @@ -258,7 +258,7 @@ build-push-image-substrate-pr:
# GIT_STRATEGY: none
# DOCKER_USER: ${PARITYPR_USER}
# DOCKER_PASS: ${PARITYPR_PASS}
# # scripts/ci/dockerfiles/malus_injected.Dockerfile
# # docker/dockerfiles/malus_injected.Dockerfile
# DOCKERFILE: malus_injected.Dockerfile
# IMAGE_NAME: docker.io/paritypr/malus
# needs:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ LABEL io.parity.image.authors=${AUTHORS} \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="${DOC_URL}" \
io.parity.image.description="${DESCRIPTION}" \
io.parity.image.source="https://github.com/paritytech/polkadot-sdk/blob/${VCS_REF}/docker/scripts/binary_injected.Dockerfile"
io.parity.image.source="https://github.com/paritytech/polkadot-sdk/blob/${VCS_REF}/docker/dockerfiles/binary_injected.Dockerfile"

USER root
WORKDIR /app
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.title="${IMAGE_NAME}" \
io.parity.image.description="Injected adder-collator Docker image" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/scripts/ci/dockerfiles/collator_injected.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/dockerfiles/collator_injected.Dockerfile" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="https://github.com/paritytech/polkadot/"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ services:
genesis_state:
build:
context: .
dockerfile: ./docker/test-parachain-collator.dockerfile
dockerfile: ./docker/dockerfiles/test-parachain-collator.dockerfile
image: "ctpc:latest"
volumes:
- "genesis-state:/data"
Expand All @@ -73,7 +73,7 @@ services:
collator:
build:
context: .
dockerfile: ./docker/test-parachain-collator.dockerfile
dockerfile: ./docker/dockerfiles/test-parachain-collator.dockerfile
target: collator
image: "ctpc:collator"
volumes:
Expand All @@ -90,7 +90,7 @@ services:
runtime:
build:
context: .
dockerfile: ./docker/test-parachain-collator.dockerfile
dockerfile: ./docker/dockerfiles/test-parachain-collator.dockerfile
target: runtime
image: "ctpc:runtime"
volumes:
Expand All @@ -100,7 +100,7 @@ services:
registrar:
build:
context: .
dockerfile: ./docker/parachain-registrar.dockerfile
dockerfile: ./docker/dockerfiles/parachain-registrar.dockerfile
image: para-reg:latest
volumes:
- "genesis-state:/genesis"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ CMD [ "--version" ]

# To use the pjs build stage to access the blockchain from the host machine:
#
# docker build -f docker/parachain-registrar.dockerfile --target pjs -t parachain-registrar:pjs .
# docker build -f docker/dockerfiles/parachain-registrar.dockerfile --target pjs -t parachain-registrar:pjs .
# alias pjs='docker run --rm --net cumulus_testing_net parachain-registrar:pjs --ws ws://172.28.1.1:9944'
#
# Then, as long as the chain is running, you can use the polkadot-js-api CLI like:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.title="${IMAGE_NAME}" \
io.parity.image.description="Cumulus, the Polkadot collator." \
io.parity.image.source="https://github.com/paritytech/cumulus/blob/${VCS_REF}/docker/polkadot-parachain/docker/polkadot-parachain-debug_unsigned_injected.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/cumulus/blob/${VCS_REF}/docker/dockerfiles/polkadot-parachain/polkadot-parachain-debug_unsigned_injected.Dockerfile" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="https://github.com/paritytech/cumulus/"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# This file is sourced from https://github.com/paritytech/polkadot/blob/master/docker/polkadot/polkadot_builder.Dockerfile
# This file is sourced from https://github.com/paritytech/polkadot/blob/master/docker/dockerfiles/polkadot/polkadot_builder.Dockerfile
# This is the build stage for polkadot-parachain. Here we create the binary in a temporary image.
FROM docker.io/paritytech/ci-linux:production as builder

Expand All @@ -14,7 +14,7 @@ LABEL io.parity.image.type="builder" \
io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.description="Multistage Docker image for polkadot-parachain" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/polkadot-parachain/polkadot-parachain_builder.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/dockerfiles/polkadot-parachain/polkadot-parachain_builder.Dockerfile" \
io.parity.image.documentation="https://github.com/paritytech/cumulus"

COPY --from=builder /cumulus/target/release/polkadot-parachain /usr/local/bin
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.title="${IMAGE_NAME}" \
io.parity.image.description="Cumulus, the Polkadot collator." \
io.parity.image.source="https://github.com/paritytech/polkadot-sdk/blob/${VCS_REF}/docker/polkadot-parachain/polkadot-parachain_injected.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/polkadot-sdk/blob/${VCS_REF}/docker/dockerfiles/polkadot-parachain/polkadot-parachain_injected.Dockerfile" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="https://github.com/paritytech/polkadot-sdk/"
Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ LABEL description="Multistage Docker image for Polkadot: a platform for web3" \
io.parity.image.authors="chevdor@gmail.com, devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.description="Polkadot: a platform for web3" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/polkadot/polkadot_builder.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/dockerfiles/polkadot/polkadot_builder.Dockerfile" \
io.parity.image.documentation="https://github.com/paritytech/polkadot/"

COPY --from=builder /polkadot/target/release/polkadot /usr/local/bin
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.title="${IMAGE_NAME}" \
io.parity.image.description="Polkadot: a platform for web3" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/polkadot/polkadot_injected_debug.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/dockerfiles/polkadot/polkadot_injected_debug.Dockerfile" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="https://github.com/paritytech/polkadot/"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.title="parity/polkadot" \
io.parity.image.description="Polkadot: a platform for web3. This is the official Parity image with an injected binary." \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/polkadot/polkadot_injected_release.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/polkadot/blob/${VCS_REF}/docker/dockerfiles/polkadot/polkadot_injected_release.Dockerfile" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="https://github.com/paritytech/polkadot/"
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# This file is sourced from https://github.com/paritytech/polkadot/blob/master/docker/polkadot/polkadot_builder.Dockerfile
# This file is sourced from https://github.com/paritytech/polkadot/blob/master/docker/dockerfiles/polkadot/polkadot_builder.Dockerfile
FROM docker.io/paritytech/ci-linux:production as builder

WORKDIR /cumulus
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ LABEL io.parity.image.authors="devops-team@parity.io" \
io.parity.image.vendor="Parity Technologies" \
io.parity.image.title="${IMAGE_NAME}" \
io.parity.image.description="Test parachain for Zombienet" \
io.parity.image.source="https://github.com/paritytech/cumulus/blob/${VCS_REF}/docker/test-parachain_injected.Dockerfile" \
io.parity.image.source="https://github.com/paritytech/cumulus/blob/${VCS_REF}/docker/dockerfiles/test-parachain_injected.Dockerfile" \
io.parity.image.revision="${VCS_REF}" \
io.parity.image.created="${BUILD_DATE}" \
io.parity.image.documentation="https://github.com/paritytech/cumulus/"
Expand Down
2 changes: 1 addition & 1 deletion docker/scripts/build-injected.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ REGISTRY=${REGISTRY:-docker.io}

# The following line ensure we know the project root
PROJECT_ROOT=${PROJECT_ROOT:-$(git rev-parse --show-toplevel)}
DOCKERFILE=${DOCKERFILE:-docker/binary_injected.Dockerfile}
DOCKERFILE=${DOCKERFILE:-docker/dockerfiles/binary_injected.Dockerfile}
VERSION_TOML=$(grep "^version " $PROJECT_ROOT/Cargo.toml | grep -oE "([0-9\.]+-?[0-9]+)")

#n The following VAR have default that can be overriden
Expand Down
2 changes: 1 addition & 1 deletion docker/scripts/polkadot-parachain/build-injected.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ PROJECT_ROOT=`git rev-parse --show-toplevel`

export BINARY=polkadot-parachain
export ARTIFACTS_FOLDER=$1
export DOCKERFILE="docker/polkadot-parachain/polkadot-parachain_injected.Dockerfile"
export DOCKERFILE="docker/dockerfiles/polkadot-parachain/polkadot-parachain_injected.Dockerfile"
# export TAGS=...

$PROJECT_ROOT/docker/scripts/build-injected.sh
2 changes: 1 addition & 1 deletion docker/scripts/polkadot-parachain_build-injected-image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@ IMAGE_NAME=${IMAGE_NAME:-polkadot-parachain}
docker build --no-cache \
--build-arg IMAGE_NAME=$IMAGE_NAME \
-t $OWNER/$IMAGE_NAME \
-f ./docker/polkadot-parachain/polkadot-parachain_injected.Dockerfile \
-f ./docker/dockerfiles/polkadot-parachain/polkadot-parachain_injected.Dockerfile \
. && docker images
2 changes: 1 addition & 1 deletion docker/scripts/staking-miner/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,4 +10,4 @@ ENGINE=podman

echo "Building the staking-miner using the Builder image"
echo "PROJECT_ROOT=$PROJECT_ROOT"
$ENGINE build -t staking-miner -f staking-miner_builder.Dockerfile "$PROJECT_ROOT"
$ENGINE build -t staking-miner -f "${PROJECT_ROOT}/docker/dockerfiles/staking-miner/staking-miner_builder.Dockerfile" "$PROJECT_ROOT"
2 changes: 1 addition & 1 deletion docs/container.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ anyone to get a working container image without requiring any of the Rust toolch
```bash
docker build \
--tag $OWNER/$IMAGE_NAME \
--file ./docker/polkadot-parachain/polkadot-parachain_builder.Containerfile .
--file ./docker/dockerfiles/polkadot-parachain/polkadot-parachain_builder.Containerfile .
```

You may then run your new container:
Expand Down
2 changes: 1 addition & 1 deletion docs/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ There are 3 options to build a Polkadot container image:
To get up and running with the smallest footprint on your system, you may use an existing Polkadot Container image.

You may also build a Polkadot container image yourself (it takes a while...) using the container specs
`docker/polkadot/polkadot_builder.Dockerfile`.
`docker/dockerfiles/polkadot/polkadot_builder.Dockerfile`.

### Debian injected

Expand Down
2 changes: 1 addition & 1 deletion polkadot/utils/staking-miner/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ to be installed. The trade-off however is that it takes a little longer to build
tasks. You may build the multi-stage image the root of the Polkadot repository with:
```
TODO: UPDATE THAT
docker build -t staking-miner -f scripts/ci/dockerfiles/staking-miner/staking-miner_builder.Dockerfile .
docker build -t staking-miner -f docker/dockerfiles/staking-miner/staking-miner_builder.Dockerfile .
```

### Running
Expand Down