Container to wrap tooling and generate package #29

castelblanque · 2022-08-24T09:04:50Z

This PR wraps most of the tooling needed to generate the Carvel package into a container.

For whoever needs to publish a new Kubeapps Carvel package, it is not needed anymore to have locally:

Carvel toolset
Kind
Helm
Bitnami's Readme generator
yq processor

Plus it will allow us to automate it, adding it in a CI/GitHub actions pipeline so that the only needed parameter to generate a package will be the desired version.

Only logic part left outside of the container is the actual Git committing and release generation in GitHub, because it needs user identification and authentication. This can be improved later and added into the container logic completely.

Resolves #11

antgamdia

Great improvements, thanks!

antgamdia · 2022-08-24T09:28:13Z

Dockerfile

+# Copyright 2022 the Kubeapps contributors.
+# SPDX-License-Identifier: Apache-2.0
+
+FROM node:bullseye


We better use a BAC image instead with no rolling tags for reproducibility, like FROM bitnami/node:16.16.0, no?

antgamdia · 2022-08-24T09:29:24Z

Dockerfile

+WORKDIR /usr/src
+
+# Install Bitnami's Readme generator
+RUN git clone --single-branch --no-tags https://github.com/bitnami-labs/readme-generator-for-helm readmenator


Suggested change

RUN git clone --single-branch --no-tags https://github.com/bitnami-labs/readme-generator-for-helm readmenator

RUN git clone --single-branch --depth 1 --no-tags https://github.com/bitnami-labs/readme-generator-for-helm readmenator

antgamdia · 2022-08-24T09:31:10Z

Dockerfile

+# Install Bitnami's Readme generator
+RUN git clone --single-branch --no-tags https://github.com/bitnami-labs/readme-generator-for-helm readmenator
+RUN cd ./readmenator && npm install
+RUN npm install --global --only=production ./readmenator


We may want to use the npm ci command instead to honor the package-lock.json ?

Oddly enough, it does not work for global packages. Will take a look in the future combining ci with the global modifier.

antgamdia · 2022-08-24T09:31:27Z

Dockerfile

+RUN npm install --global --only=production ./readmenator
+
+# Install Carvel toolset
+RUN bash -c "set -eo pipefail; wget -O- https://carvel.dev/install.sh | bash"


Does it install kapp-ctrl CLI as well?

These are the binaries installed:

kctrl

ytt

imgpkg

kbld

kapp

kwt

vendir

antgamdia · 2022-08-24T09:33:34Z

Dockerfile

+RUN curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | tee /usr/share/keyrings/helm.gpg > /dev/null
+RUN echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | \
+        tee /etc/apt/sources.list.d/helm-stable-debian.list
+RUN apt-get update && apt-get install --yes helm


What about relying on the helm binary directly instead of the OS packaging system? I mean, for having control of which version we are installing, like:

wget https://get.helm.sh/helm-${HELM_VERSION}-linux-amd64.tar.gz tar zxf helm-$HELM_VERSION-linux-amd64.tar.gz sudo mv linux-amd64/helm /usr/local/bin/

Agree, will add it.

antgamdia · 2022-08-24T09:36:09Z

Dockerfile

+# Install Docker
+RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
+RUN add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/debian bullseye stable"
+RUN apt-get update && apt-get install --yes docker-ce docker-ce-cli containerd.io


also, for reproducibility, perhaps setting sticking to a version is useful, like:
(don't know which versions, though)

Suggested change

RUN apt-get update && apt-get install --yes docker-ce docker-ce-cli containerd.io

RUN apt-get update && apt-get install --yes docker-ce=x.y.z docker-ce-cli=x.y.z containerd.io=x.y.z

antgamdia · 2022-08-24T09:37:26Z

Dockerfile

+RUN curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
+RUN echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | \
+        tee /etc/apt/sources.list.d/kubernetes.list
+RUN apt-get update && apt-get install --yes kubectl


Idem with the version:

Suggested change

RUN apt-get update && apt-get install --yes kubectl

RUN apt-get update && apt-get install --yes kubectl=x.y.z

antgamdia · 2022-08-24T09:37:51Z

Dockerfile

+# Install Kind
+RUN wget "https://kind.sigs.k8s.io/dl/$KIND_VERSION/kind-$(uname)-$(dpkg --print-architecture)" -O /usr/local/bin/kind
+RUN chmod +x /usr/local/bin/kind
+RUN kind --version


Why adding here a check whereas in other binaries we are not doing it?

Oops, development leftover. Will remove.

antgamdia · 2022-08-24T09:38:50Z

Dockerfile

+RUN apt-get update && apt-get install --yes kubectl
+
+# Install Tanzu CLI
+RUN wget "https://github.com/vmware-tanzu/tanzu-framework/releases/download/${TANZU_FW_VERSION}/tanzu-cli-linux-amd64.tar.gz"


What the fw in TANZU_FW_VERSION stands for? Why not TANZU_CLI_VERSION? Not really important, though

It stands for "framework", which comes from the actual name of the project: tanzu-framework. I'll rename it to TANZU_FRAMEWORK_VERSION.

antgamdia · 2022-08-24T09:43:15Z

release-kubeapps-package.sh

+  git push "$git_origin" "tags/$tag"
+
+  echo "Creating release for $tag"
+  gh release create "$tag" "./$version_with_suffix/package.yaml" "./metadata.yaml" "./README.md" \


Should we be installing the GitHub CLI as well in the container?

I've added a comment about it in this PR's description. For the Git+GitHub operations we need valid credentials, which makes things more complex. I decided to have it mostly automated, except this part, that can be improved in a later PR.

Ah, ok... but can't we just run the container and pass through it some env vars with the credentials docker run ... -e XXXX=YYY ? Anyway, ok with the semi-automated way for now.

can't we just run the container and pass through it some env vars with the credentials

Exactly, that will be done in a later PR. I've added a comment in the issue and will leave it opened.

antgamdia · 2022-08-24T09:50:55Z

release-kubeapps-package.sh

+  local version_with_suffix="$version$packaging_version_suffix"
+  local tag="v$version_with_suffix"
+  git add "./$version_with_suffix"
+  git commit -m "Adding $tag files"


Should we also add the -s and -S flags to DCO and GPG-sign the commits and tags?

Agree, added issue #31 to do it in a separate PR.

Rafa Castelblanque added 3 commits August 24, 2022 09:56

Wrap package generation in a container

ef271b3

Some final adjustments for scripts

983d642

Added documentation

4bcba24

vmwclabot added the cla-not-required label Aug 24, 2022

Removed not needed parameter

b87cec6

castelblanque requested a review from antgamdia August 24, 2022 09:07

antgamdia approved these changes Aug 24, 2022

View reviewed changes

antgamdia reviewed Aug 24, 2022

View reviewed changes

castelblanque mentioned this pull request Aug 24, 2022

Sign off and GPG-sign Git commits when releasing a package #31

Open

PR review changes

3745552

castelblanque merged commit fff09d3 into main Aug 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Container to wrap tooling and generate package #29

Container to wrap tooling and generate package #29

castelblanque commented Aug 24, 2022

antgamdia left a comment

antgamdia Aug 24, 2022

antgamdia Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

antgamdia Aug 24, 2022

antgamdia Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

antgamdia Aug 24, 2022

castelblanque Aug 24, 2022

	RUN git clone --single-branch --no-tags https://github.com/bitnami-labs/readme-generator-for-helm readmenator
	RUN git clone --single-branch --depth 1 --no-tags https://github.com/bitnami-labs/readme-generator-for-helm readmenator

	RUN apt-get update && apt-get install --yes docker-ce docker-ce-cli containerd.io
	RUN apt-get update && apt-get install --yes docker-ce=x.y.z docker-ce-cli=x.y.z containerd.io=x.y.z

Container to wrap tooling and generate package #29

Container to wrap tooling and generate package #29

Conversation

castelblanque commented Aug 24, 2022

antgamdia left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment