-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Packaging] Use CentOS Stream 8 to build el8
RPM package
#20918
Changes from 11 commits
74c2205
521ceea
756e664
36d3389
ec44277
3ca5d20
7b5a07e
340c4be
0b6287c
e88e373
7a6e98c
5050e7e
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -700,8 +700,8 @@ jobs: | |
displayName: 'Test Homebrew Package' | ||
|
||
|
||
- job: BuildYumPackageMariner | ||
displayName: Build Yum Package Mariner | ||
- job: BuildRpmPackageMariner | ||
displayName: Build Rpm Package Mariner | ||
pool: | ||
vmImage: 'ubuntu-20.04' | ||
steps: | ||
|
@@ -717,14 +717,16 @@ jobs: | |
BuildDropPath: $(Build.ArtifactStagingDirectory) | ||
|
||
- task: PublishPipelineArtifact@0 | ||
displayName: 'Publish Artifact: yum-mariner' | ||
displayName: 'Publish Artifact: rpm-mariner' | ||
inputs: | ||
TargetPath: $(Build.ArtifactStagingDirectory) | ||
ArtifactName: yum-mariner | ||
ArtifactName: rpm-mariner | ||
|
||
|
||
- job: BuildYumPackage | ||
displayName: Build Yum Package | ||
- job: BuildRpmPackageCentOS7 | ||
displayName: Build Rpm Package CentOS 7 | ||
# Uncomment this line to disable this job on Pull Requests | ||
# condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule')) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This line was removed by #20608. Bring it back as a comment. |
||
pool: | ||
vmImage: 'ubuntu-20.04' | ||
steps: | ||
|
@@ -733,23 +735,49 @@ jobs: | |
inputs: | ||
targetType: 'filePath' | ||
filePath: scripts/release/rpm/pipeline.sh | ||
|
||
env: | ||
IMAGE: centos7 | ||
- task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 | ||
displayName: 'SBOM' | ||
inputs: | ||
BuildDropPath: $(Build.ArtifactStagingDirectory) | ||
- task: PublishPipelineArtifact@0 | ||
displayName: 'Publish Artifact: rpm' | ||
inputs: | ||
TargetPath: $(Build.ArtifactStagingDirectory) | ||
ArtifactName: rpm-centos7 | ||
|
||
# rpmbuild on CentOS Stream 8 is slow, we use a separate job for CentOS Stream 8 instead of strategy.matrix so that | ||
# TestRpmPackage can start right after BuildRpmPackageCentOS7 finishes. | ||
- job: BuildRpmPackageCentOSStream8 | ||
displayName: Build Rpm Package CentOS Stream 8 | ||
# Do not run this job for Pull Requests due to the slowness | ||
condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule')) | ||
pool: | ||
vmImage: 'ubuntu-20.04' | ||
steps: | ||
- task: Bash@3 | ||
displayName: 'Build Rpm Package' | ||
inputs: | ||
targetType: 'filePath' | ||
filePath: scripts/release/rpm/pipeline.sh | ||
env: | ||
IMAGE: centos_stream8 | ||
- task: AzureArtifacts.manifest-generator-task.manifest-generator-task.ManifestGeneratorTask@0 | ||
displayName: 'SBOM' | ||
inputs: | ||
BuildDropPath: $(Build.ArtifactStagingDirectory) | ||
- task: PublishPipelineArtifact@0 | ||
displayName: 'Publish Artifact: yum' | ||
displayName: 'Publish Artifact: rpm' | ||
inputs: | ||
TargetPath: $(Build.ArtifactStagingDirectory) | ||
ArtifactName: yum | ||
ArtifactName: rpm-centos_stream8 | ||
|
||
|
||
- job: TestYumPackage | ||
displayName: Test Yum Package | ||
- job: TestRpmPackage | ||
displayName: Test Rpm Package | ||
timeoutInMinutes: 120 | ||
dependsOn: BuildYumPackage | ||
dependsOn: BuildRpmPackageCentOS7 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As discussed in
Here we make 2 deferent jobs:
so that |
||
condition: and(succeeded(), in(variables['Build.Reason'], 'IndividualCI', 'BatchedCI', 'Manual', 'Schedule')) | ||
pool: | ||
vmImage: 'ubuntu-20.04' | ||
|
@@ -764,24 +792,24 @@ jobs: | |
- task: DownloadPipelineArtifact@1 | ||
displayName: 'Download Build Artifacts' | ||
inputs: | ||
TargetPath: '$(Build.ArtifactStagingDirectory)/yum' | ||
artifactName: yum | ||
TargetPath: '$(Build.ArtifactStagingDirectory)/rpm' | ||
artifactName: rpm-centos7 | ||
|
||
|
||
- bash: | | ||
set -ex | ||
|
||
CLI_VERSION=`cat $SYSTEM_ARTIFACTSDIRECTORY/metadata/version` | ||
YUM_NAME=azure-cli-$CLI_VERSION-1.el7.x86_64.rpm | ||
YUM_FILE=$SYSTEM_ARTIFACTSDIRECTORY/yum/$YUM_NAME | ||
RPM_NAME=azure-cli-$CLI_VERSION-1.el7.x86_64.rpm | ||
RPM_FILE=$SYSTEM_ARTIFACTSDIRECTORY/rpm/$RPM_NAME | ||
|
||
echo "== Test yum package on CentOS ==" | ||
echo "== Test rpm package on CentOS ==" | ||
|
||
IMAGE=centos:centos7 | ||
docker pull $IMAGE | ||
docker run --rm -e YUM_NAME=$YUM_NAME -v $SYSTEM_ARTIFACTSDIRECTORY/yum:/mnt/yum -v $(pwd):/azure-cli $IMAGE /bin/bash "/azure-cli/scripts/release/rpm/test_rpm_in_docker.sh" | ||
docker run --rm -e RPM_NAME=$RPM_NAME -v $SYSTEM_ARTIFACTSDIRECTORY/rpm:/mnt/rpm -v $(pwd):/azure-cli $IMAGE /bin/bash "/azure-cli/scripts/release/rpm/test_rpm_in_docker.sh" | ||
|
||
displayName: 'Test Yum Package' | ||
displayName: 'Test Rpm Package' | ||
|
||
- job: BuildDebPackages | ||
displayName: Build Deb Packages | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# CentOS Stream 8 docker image is distributed as quay.io/centos/centos:stream8 at | ||
# https://quay.io/repository/centos/centos?tab=tags. See https://wiki.centos.org/FAQ/CentOSStream | ||
|
||
ARG tag=stream8 | ||
|
||
FROM quay.io/centos/centos:${tag} AS build-env | ||
ARG cli_version=dev | ||
|
||
RUN yum update -y | ||
RUN yum install -y wget rpm-build gcc libffi-devel python3-devel openssl-devel make bash diffutils patch dos2unix python3-virtualenv perl | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
WORKDIR /azure-cli | ||
|
||
COPY . . | ||
|
||
RUN dos2unix ./scripts/release/rpm/azure-cli.spec && \ | ||
REPO_PATH=$(pwd) CLI_VERSION=$cli_version rpmbuild -v -bb --clean scripts/release/rpm/azure-cli.spec && \ | ||
cp /root/rpmbuild/RPMS/x86_64/azure-cli-${cli_version}-1.*.x86_64.rpm /azure-cli-dev.rpm | ||
|
||
FROM quay.io/centos/centos:${tag} AS execution-env | ||
|
||
RUN yum update -y | ||
RUN yum install -y python3 python3-virtualenv | ||
|
||
COPY --from=build-env /azure-cli-dev.rpm ./ | ||
RUN rpm -i ./azure-cli-dev.rpm && \ | ||
az --version |
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -6,34 +6,38 @@ set -exv | |
|
||
: "${BUILD_STAGINGDIRECTORY:?BUILD_STAGINGDIRECTORY environment variable not set.}" | ||
|
||
# IMAGE should be one of 'centos7', 'centos_stream8' | ||
: "${IMAGE:?IMAGE environment variable not set.}" | ||
|
||
CLI_VERSION=`cat src/azure-cli/azure/cli/__main__.py | grep __version__ | sed s/' '//g | sed s/'__version__='// | sed s/\"//g` | ||
|
||
# Create a container image that includes the source code and a built RPM using this file. | ||
docker build \ | ||
--target build-env \ | ||
--build-arg cli_version=${CLI_VERSION} \ | ||
-f ./scripts/release/rpm/Dockerfile.centos \ | ||
-t azure/azure-cli:centos7-builder \ | ||
-f ./scripts/release/rpm/Dockerfile.${IMAGE} \ | ||
-t azure/azure-cli:${IMAGE}-builder \ | ||
. | ||
|
||
# Continue the previous build, and create a container that has the current azure-cli build but not the source code. | ||
docker build \ | ||
--build-arg cli_version=${CLI_VERSION} \ | ||
-f ./scripts/release/rpm/Dockerfile.centos \ | ||
-t azure/azure-cli:centos7 \ | ||
-f ./scripts/release/rpm/Dockerfile.${IMAGE} \ | ||
-t azure/azure-cli:${IMAGE} \ | ||
. | ||
|
||
# Extract the built RPM so that it can be distributed independently. | ||
docker run \ | ||
azure/azure-cli:centos7-builder \ | ||
cat /root/rpmbuild/RPMS/x86_64/azure-cli-${CLI_VERSION}-1.el7.x86_64.rpm \ | ||
> ${BUILD_STAGINGDIRECTORY}/azure-cli-${CLI_VERSION}-1.el7.x86_64.rpm | ||
# The RPM file looks like azure-cli-2.32.0-1.el7.x86_64.rpm | ||
id=$(docker create azure/azure-cli:${IMAGE}-builder) | ||
# https://docs.docker.com/engine/reference/commandline/cp/ | ||
# Append /. so that the x86_64 folder's content is copied, instead of x86_64 folder itself. | ||
docker cp $id:/root/rpmbuild/RPMS/x86_64/. ${BUILD_STAGINGDIRECTORY} | ||
|
||
# Save these too a staging directory so that later build phases can choose to save them as Artifacts or publish them to | ||
# a registry. | ||
# | ||
# The products of `docker save` can be rehydrated using `docker load`. | ||
mkdir -p ${BUILD_STAGINGDIRECTORY}/docker | ||
docker save azure/azure-cli:centos7-builder | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_centos7-builder.tar.gz & | ||
docker save azure/azure-cli:centos7 | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_centos7.tar.gz & | ||
wait | ||
# mkdir -p ${BUILD_STAGINGDIRECTORY}/docker | ||
# docker save azure/azure-cli:${IMAGE}-builder | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_${IMAGE}-builder.tar.gz & | ||
# docker save azure/azure-cli:${IMAGE} | gzip > ${BUILD_STAGINGDIRECTORY}/docker/azure_azure-cli_${IMAGE}.tar.gz & | ||
# wait | ||
Comment on lines
+40
to
+43
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually we never rehydrated these files. We disable them to save space on ADO. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
RPM is the actual package and YUM is the package manager (see https://packages.microsoft.com/yumrepos/azure-cli/). YUM has been replaced by DNF in many modern systems. So RPM should be the correct name here, like DEB (package) vs APT (package manager).