Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Packaging] Use CentOS Stream 8 to build el8 RPM package #20918

Merged
merged 12 commits into from
Mar 15, 2022
Merged

[Packaging] Use CentOS Stream 8 to build el8 RPM package #20918

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
74c2205
el8
jiasli Jan 6, 2022
521ceea
update
jiasli Jan 6, 2022
756e664
Merge remote-tracking branch 'azure/dev' into el8
jiasli Feb 9, 2022
36d3389
Add comment
jiasli Feb 9, 2022
ec44277
comment out docker save
jiasli Feb 9, 2022
3ca5d20
Rename yum to rpm
jiasli Feb 9, 2022
7b5a07e
Merge remote-tracking branch 'azure/dev' into el8
jiasli Feb 9, 2022
340c4be
Do not use matrix
jiasli Feb 9, 2022
0b6287c
Update azure-pipelines.yml
jiasli Feb 9, 2022
e88e373
Merge remote-tracking branch 'azure/dev' into el8
jiasli Mar 11, 2022
7a6e98c
update
jiasli Mar 11, 2022
5050e7e
Update test_rpm_in_docker.sh
jiasli Mar 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 47 additions & 19 deletions azure-pipelines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -700,8 +700,8 @@ jobs:
displayName: 'Test Homebrew Package'


- job: BuildYumPackageMariner
displayName: Build Yum Package Mariner
- job: BuildRpmPackageMariner
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RPM is the actual package and YUM is the package manager (see https://packages.microsoft.com/yumrepos/azure-cli/). YUM has been replaced by DNF in many modern systems. So RPM should be the correct name here, like DEB (package) vs APT (package manager).

displayName: Build Rpm Package Mariner
pool:
vmImage: 'ubuntu-20.04'
steps:
Expand All @@ -717,14 +717,16 @@ jobs:
BuildDropPath: $(Build.ArtifactStagingDirectory)

- task: PublishPipelineArtifact@0
displayName: 'Publish Artifact: yum-mariner'
displayName: 'Publish Artifact: rpm-mariner'
inputs:
TargetPath: $(Build.ArtifactStagingDirectory)
ArtifactName: yum-mariner
ArtifactName: rpm-mariner


- job: BuildYumPackage
displayName: Build Yum Package
- job: BuildRpmPackageCentOS7
displayName: Build Rpm Package CentOS 7
# Uncomment this line to disable this job on Pull Requests
# condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule'))
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line was removed by #20608. Bring it back as a comment.

pool:
vmImage: 'ubuntu-20.04'
steps:
Expand All @@ -733,23 +735,49 @@ jobs:
inputs:
targetType: 'filePath'
filePath: scripts/release/rpm/pipeline.sh

env:
IMAGE: centos7
- task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0
displayName: 'SBOM'
inputs:
BuildDropPath: $(Build.ArtifactStagingDirectory)
- task: PublishPipelineArtifact@0
displayName: 'Publish Artifact: rpm'
inputs:
TargetPath: $(Build.ArtifactStagingDirectory)
ArtifactName: rpm-centos7

# rpmbuild on CentOS Stream 8 is slow, we use a separate job for CentOS Stream 8 instead of strategy.matrix so that
# TestRpmPackage can start right after BuildRpmPackageCentOS7 finishes.
- job: BuildRpmPackageCentOSStream8
displayName: Build Rpm Package CentOS Stream 8
# Do not run this job for Pull Requests due to the slowness
condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule'))
pool:
vmImage: 'ubuntu-20.04'
steps:
- task: Bash@3
displayName: 'Build Rpm Package'
inputs:
targetType: 'filePath'
filePath: scripts/release/rpm/pipeline.sh
env:
IMAGE: centos_stream8
- task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0
displayName: 'SBOM'
inputs:
BuildDropPath: $(Build.ArtifactStagingDirectory)
- task: PublishPipelineArtifact@0
displayName: 'Publish Artifact: yum'
displayName: 'Publish Artifact: rpm'
inputs:
TargetPath: $(Build.ArtifactStagingDirectory)
ArtifactName: yum
ArtifactName: rpm-centos_stream8


- job: TestYumPackage
displayName: Test Yum Package
- job: TestRpmPackage
displayName: Test Rpm Package
timeoutInMinutes: 120
dependsOn: BuildYumPackage
dependsOn: BuildRpmPackageCentOS7
Copy link
Member Author

@jiasli jiasli Feb 9, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As discussed in

dependsOn only works for the base job (all matrixes), and it can't work on a specific matrix. Since CentOS Stream 8's rpm builder is extremely slow for some reason, TestRpmPackage will wait for it to finish, making the whole pipeline much slower.

Here we make 2 deferent jobs:

  • BuildRpmPackageCentOS7
  • BuildRpmPackageCentOSStream8

so that TestRpmPackage can start right after BuildRpmPackageCentOS7 finishes, like before.

condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule'))
pool:
vmImage: 'ubuntu-20.04'
Expand All @@ -764,24 +792,24 @@ jobs:
- task: DownloadPipelineArtifact@1
displayName: 'Download Build Artifacts'
inputs:
TargetPath: '$(Build.ArtifactStagingDirectory)/yum'
artifactName: yum
TargetPath: '$(Build.ArtifactStagingDirectory)/rpm'
artifactName: rpm-centos7


- bash: |
set -ex

CLI_VERSION=`cat $SYSTEM_ARTIFACTSDIRECTORY/metadata/version`
YUM_NAME=azure-cli-$CLI_VERSION-1.el7.x86_64.rpm
YUM_FILE=$SYSTEM_ARTIFACTSDIRECTORY/yum/$YUM_NAME
RPM_NAME=azure-cli-$CLI_VERSION-1.el7.x86_64.rpm
RPM_FILE=$SYSTEM_ARTIFACTSDIRECTORY/rpm/$RPM_NAME

echo "== Test yum package on CentOS =="
echo "== Test rpm package on CentOS =="

IMAGE=centos:centos7
docker pull $IMAGE
docker run --rm -e YUM_NAME=$YUM_NAME -v $SYSTEM_ARTIFACTSDIRECTORY/yum:/mnt/yum -v $(pwd):/azure-cli $IMAGE /bin/bash "/azure-cli/scripts/release/rpm/test_rpm_in_docker.sh"
docker run --rm -e RPM_NAME=$RPM_NAME -v $SYSTEM_ARTIFACTSDIRECTORY/rpm:/mnt/rpm -v $(pwd):/azure-cli $IMAGE /bin/bash "/azure-cli/scripts/release/rpm/test_rpm_in_docker.sh"

displayName: 'Test Yum Package'
displayName: 'Test Rpm Package'

- job: BuildDebPackages
displayName: Build Deb Packages
Expand Down
0 scripts/release/rpm/Dockerfile.centos → scripts/release/rpm/Dockerfile.centos7
View file
Open in desktop
File renamed without changes.
27 changes: 27 additions & 0 deletions scripts/release/rpm/Dockerfile.centos_stream8
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# CentOS Stream 8 docker image is distributed as quay.io/centos/centos:stream8 at
# https://quay.io/repository/centos/centos?tab=tags. See https://wiki.centos.org/FAQ/CentOSStream

ARG tag=stream8

FROM quay.io/centos/centos:${tag} AS build-env
ARG cli_version=dev

RUN yum update -y
RUN yum install -y wget rpm-build gcc libffi-devel python3-devel openssl-devel make bash diffutils patch dos2unix python3-virtualenv perl
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

coreutils is not compatible with preinstalled coreutils-single, so we don't need to install coreutils.

# dnf list installed | grep coreutils
coreutils-single.x86_64                       8.30-12.el8                     @baseos

# yum install -y coreutils
Last metadata expiration check: 0:13:00 ago on Thu Jan  6 09:04:58 2022.
Error:
 Problem: problem with installed package coreutils-single-8.30-12.el8.x86_64
  - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-12.el8.x86_64
  - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-10.el8.x86_64
  - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-8.el8.x86_64
  - package coreutils-8.30-12.el8.x86_64 conflicts with coreutils-single provided by coreutils-single-8.30-9.el8.x86_64
  - cannot install the best candidate for the job
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

Copy link
Member Author

@jiasli jiasli Jan 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perl is not installed by rpm-build on CentOS Stream 8, so we should manually install it.


WORKDIR /azure-cli

COPY . .

RUN dos2unix ./scripts/release/rpm/azure-cli.spec && \
REPO_PATH=$(pwd) CLI_VERSION=$cli_version rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec && \
cp /root/rpmbuild/RPMS/x86_64/azure-cli-${cli_version}-1.*.x86_64.rpm /azure-cli-dev.rpm

FROM quay.io/centos/centos:${tag} AS execution-env

RUN yum update -y
RUN yum install -y python3 python3-virtualenv

COPY --from=build-env /azure-cli-dev.rpm ./
RUN rpm -i ./azure-cli-dev.rpm && \
az --version
28 changes: 16 additions & 12 deletions scripts/release/rpm/pipeline.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,34 +6,38 @@ set -exv

: "${BUILD_STAGINGDIRECTORY:?BUILD_STAGINGDIRECTORY environment variable not set.}"

# IMAGE should be one of 'centos7', 'centos_stream8'
: "${IMAGE:?IMAGE environment variable not set.}"

CLI_VERSION=`cat src/azure-cli/azure/cli/__main__.py | grep __version__ | sed s/' '//g | sed s/'__version__='// | sed s/\"//g`

# Create a container image that includes the source code and a built RPM using this file.
docker build \
--target build-env \
--build-arg cli_version=${CLI_VERSION} \
-f ./scripts/release/rpm/Dockerfile.centos \
-t azure/azure-cli:centos7-builder \
-f ./scripts/release/rpm/Dockerfile.${IMAGE} \
-t azure/azure-cli:${IMAGE}-builder \
.

# Continue the previous build, and create a container that has the current azure-cli build but not the source code.
docker build \
--build-arg cli_version=${CLI_VERSION} \
-f ./scripts/release/rpm/Dockerfile.centos \
-t azure/azure-cli:centos7 \
-f ./scripts/release/rpm/Dockerfile.${IMAGE} \
-t azure/azure-cli:${IMAGE} \
.

# Extract the built RPM so that it can be distributed independently.
docker run \
azure/azure-cli:centos7-builder \
cat /root/rpmbuild/RPMS/x86_64/azure-cli-${CLI_VERSION}-1.el7.x86_64.rpm \
> ${BUILD_STAGINGDIRECTORY}/azure-cli-${CLI_VERSION}-1.el7.x86_64.rpm
# The RPM file looks like azure-cli-2.32.0-1.el7.x86_64.rpm
id=$(docker create azure/azure-cli:${IMAGE}-builder)
# https://docs.docker.com/engine/reference/commandline/cp/
# Append /. so that the x86_64 folder's content is copied, instead of x86_64 folder itself.
docker cp $id:/root/rpmbuild/RPMS/x86_64/. ${BUILD_STAGINGDIRECTORY}

# Save these too a staging directory so that later build phases can choose to save them as Artifacts or publish them to
# a registry.
#
# The products of `docker save` can be rehydrated using `docker load`.
mkdir -p ${BUILD_STAGINGDIRECTORY}/docker
docker save azure/azure-cli:centos7-builder | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_centos7-builder.tar.gz &
docker save azure/azure-cli:centos7 | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_centos7.tar.gz &
wait
# mkdir -p ${BUILD_STAGINGDIRECTORY}/docker
# docker save azure/azure-cli:${IMAGE}-builder | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_${IMAGE}-builder.tar.gz &
# docker save azure/azure-cli:${IMAGE} | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_${IMAGE}.tar.gz &
# wait
Comment on lines +40 to +43
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually we never rehydrated these files. We disable them to save space on ADO.