GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

raulcd · 2025-09-19T09:33:13Z

TBD

Rationale for this change

What changes are included in this PR?

Move check_labels and report_ci jobs to independent reusable workflows.
Update cpp_extra to use those.
Create new linux_packaging.yml workflow replicating work that was done on crossbow. Integrate that workflow with check_labels and report_ci
Update release binary submit and binary download to run workflow and download the artifacts from the job instead of from the crossbow repository.

Are these changes tested?

Some via CI on fork and some manual testing.

Are there any user-facing changes?

No

GitHub Issue: [CI][Crossbow][Packaging] Use apache/arrow for linux-packaging jobs #47582

github-actions · 2025-09-19T09:33:41Z

⚠️ GitHub issue #47582 has been automatically assigned in GitHub to PR creator.

raulcd

Hi @kou I've started working on this and I think I have a working base that could work. I wanted to discuss it with you before moving more packages to validate we are all ok with the approach.
Let me know your thoughts. I have a couple of questions to discuss too, let me know what you think on those too.

.github/workflows/package_linux.yml

dev/release/04-binary-download.sh

raulcd · 2025-09-19T09:49:22Z

.github/workflows/cpp_extra.yml

+    uses: ./.github/workflows/check_labels.yml
+    secrets: inherit
+    with:
+      label: "CI: Extra"


I'd like to move these labels to either:

CI Extra: cpp

CI Extra: CPP

CI: cpp

CI: CPP

So we can have the others the same like:

CI Extra: linux-packaging

CI Extra: Linux Packaging

CI: linux-packaging

CI: Linux Packaging

What do you think?

I like CI: Extra: C++/CI: Extra: Package: Linux or CI: cpp/CI: package: linux/CI: package: linux: apt.

kou

Thanks!

.github/workflows/check_labels.yml

kou · 2025-09-19T12:07:36Z

.github/workflows/cpp_extra.yml

+    uses: ./.github/workflows/check_labels.yml
+    secrets: inherit
+    with:
+      label: "CI: Extra"


I like CI: Extra: C++/CI: Extra: Package: Linux or CI: cpp/CI: package: linux/CI: package: linux: apt.

.github/workflows/linux_packaging.yml

.github/workflows/package_linux.yml

.github/workflows/linux_packaging.yml

dev/release/04-binary-download.sh

raulcd

I've gone over all the comments and I think this is ready for another review.

raulcd · 2025-09-23T10:04:27Z

dev/release/binary-task.rb

-            # apache/arrow-adbc uses debian-bookworm. So the following
-            # glob must much both of them.


is adbc still using this binary tasks? in that case we probably want to maintain the previous behaviour for the task too?

Yes.

But I can change ADBC to simplify this script.

raulcd · 2025-09-23T10:07:27Z

dev/release/binary-task.rb

-            # glob must much both of them.
-            Dir.glob("#{source_dir_prefix}*/*") do |path|
-              base_name = File.basename(path)
+            destination_prefix = "#{incoming_dir}/#{distribution}"


@kou I've never done anything on ruby (I know, I should learn a little 😃 ). I've tested these tasks locally (apt:rc:copy and yum:rc:copy) with a small integration test calling rake directly and seems to be working as expected but I would appreciate some help on those.

Thanks for writing Ruby!

Sure. I can help. Can I push something to this branch if it's needed?

BTW, destination_dir is better for this variable name because this is a directory name not a prefix of path. (In this code, "prefix" is used for abc for abcde directory and "dir" is used for abcde directory.)

you can push to this branch if needed! that's completely fine with me. I am trying to simplify the folder structure as you suggested so the binary tasks are even simpler. Thanks, it's good that I am learning a little bit more about our linux packages so I can help you maintain those in the future.

kou · 2025-09-23T12:15:22Z

dev/release/binary-task.rb

-            # apache/arrow-adbc uses debian-bookworm. So the following
-            # glob must much both of them.


Yes.

But I can change ADBC to simplify this script.

kou · 2025-09-23T12:36:29Z

.github/workflows/package_linux.yml

+        run: |
+          set -ex
+          pushd dev/tasks/linux-packages
+          tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories


Let's simplify paths more to simplify binary-task.rb:

Suggested change

tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories

mkdir -p artifacts

cp -a */${TASK_NAMESPACE}/repositories/* artifacts/

tar cvzf ${{ matrix.id }}.tar.gz -C artifacts .

I have tried to simplify the paths slightly more but I am finding quite difficult to understand how all tangles together. How do we generate the repositories from those rpms, how we merge these files with other existing files from the repo, how do we create the testing repositories, etcetera so I am reverting my latest changes. I am happy to leave it like this (current generated structure) and try to simplify on a future PR or if you want to guide me on how to do it or if you want to push to this branch.

kou · 2025-09-23T12:37:19Z

dev/release/04-binary-download.sh

+
+# Wait for the GitHub Workflow that creates the Linux packages
+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"


We don't need . here because all information is passed via arguments:

Suggested change

. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"

"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"

kou · 2025-09-23T12:45:37Z

dev/release/binary-task.rb

-            # glob must much both of them.
-            Dir.glob("#{source_dir_prefix}*/*") do |path|
-              base_name = File.basename(path)
+            destination_prefix = "#{incoming_dir}/#{distribution}"


Thanks for writing Ruby!

Sure. I can help. Can I push something to this branch if it's needed?

BTW, destination_dir is better for this variable name because this is a directory name not a prefix of path. (In this code, "prefix" is used for abc for abcde directory and "dir" is used for abcde directory.)

dev/release/binary-task.rb

kou · 2025-09-23T12:48:34Z

dev/release/post-05-update-gh-release-notes.sh

 # Wait for the GitHub Workflow that creates the GitHub Release
 # to finish before updating the release notes.
-"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${TAG}" "${WORKFLOW}"
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${TAG}" "${WORKFLOW}"


We don't need . here.

dev/tasks/linux-packages/helper.rb

kou · 2025-09-23T12:57:11Z

dev/release/utils-watch-gh-workflow.sh

+download_artifacts() {
+  RUN_ID=$(get_run_id)
+  echo "Downloading artitfacts for workflow with ID: ${RUN_ID}"
+  gh run download \
+    ${RUN_ID} \
+    --repo "${REPOSITORY}" \
+    --dir "$1"
+}


Hmm. Defining this function in this file isn't good because this file name is watch-gh-workflow.

We could rename the file to utils-gh-workflow.sh but if we use release for the artifacts this won't be necessary.

kou · 2025-09-23T12:57:38Z

dev/release/04-binary-download.sh

+# to finish before downloading the artifacts.
+. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
+
+RUN_ID=$(get_run_id)


Ah, you want to use a function defined in utils-watch-gh-workflow.sh.

kou · 2025-09-23T12:58:39Z

dev/release/04-binary-download.sh

+
+RUN_ID=$(get_run_id)
+# Download the artifacts created by the package_linux.yml workflow
+download_artifacts "${SOURCE_DIR}/../../packages/${CROSSBOW_JOB_ID}"


How about using GitHub Release for RC instead of GitHub Actions artifacts? If we use GitHub Release, we don't need RUN_ID: gh release download apache-arrow-${VERSION}-rc${RC} ...

so you mean, for scheduled or PR/push use artifacts but if workflow_dispatch was used, which is trigger of RC, upload to release? I don't want to create a release every time the job is executed but that could work. In that case we might use artifacts to test and upload to release if tests are successful and job type was workflow_dispatch

I want to upload artifacts to GitHub Release only when apache-arrow-${VERSION}-rc${RC} tag is pushed. Can we use on.push.tags instead of workflow_dispatch for RC? (I can provide changes for the approach.)

…e from tar.gz

…re binary tasks

…m/repositories subfolders Try fixing yum tests

This reverts commit c9c4631.

…s and yum/repositories subfolders" This reverts commit 516e0fb.

… even more binary tasks" This reverts commit 471e110.

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

* Use GitHub Release instead of GitHub Actions' artifacts * Build directory structure in package generator not binary-task.rb

kou

I've pushed changes for the suggested workflow.

I'll fix CI failures and then test release workflow by pushing a RC tag to fork repository.

kou · 2025-10-02T07:37:57Z

dev/tasks/linux-packages/helper.rb

-    def detect_repo
-      detect_env("REPO")
+    def github_repository
+      ENV["GITHUB_REPOSITORY"] || "apache/arrow"


GitHub Actions provides GITHUB_REPOSITORY environment variable by default. So this uses fork's GitHub Container registry on fork automatically.

kou · 2025-10-02T07:42:56Z

dev/tasks/linux-packages/yum/build.sh

 repository="${repositories}/${distribution}/${distribution_version}"
 rpm_dir="${repository}/${architecture}/Packages"
-srpm_dir="${repository}/source/SRPMS"
+srpm_dir="${repository}/Source/Packages"


AlmaLinux use Source/Packages: https://vault.almalinux.org/10.0/BaseOS/Source/Packages/

kou · 2025-10-02T07:44:18Z

dev/release/binary-task.rb

      ["almalinux", "8"],
      ["amazon-linux", "2023"],
      ["centos", "9-stream"],
-      ["centos", "8-stream"],


We already dropped support for CentOS Stream 8: #46953

kou · 2025-10-02T07:45:02Z

dev/release/binary-task.rb

+            Dir.glob("#{source_dir_prefix}*.tar.gz") do |tar_gz|
+              sh("tar", "xf", tar_gz, "-C", incoming_dir)
+              progress_reporter.advance
            end


We can use this for ADBC too but we need some changes in ADBC: apache/arrow-adbc#3510

kou · 2025-10-02T07:46:12Z

.github/workflows/release_candidate.yml

-      # Trigger workflow when a tag whose name matches the pattern
-      # "apache-arrow-{MAJOR}.{MINOR}.{PATCH}-rc{RC_NUM}" is pushed.
-      - "apache-arrow-[0-9]+.[0-9]+.[0-9]+-rc[0-9]+"
+      - "apache-arrow-*-rc*"


Simplify and unify tag patterns.

kou · 2025-10-02T07:48:01Z

.github/workflows/package_linux.yml

+          # We use latest .deb/.rpm of
+          # apache-arrow-apt-source/apache-arrow-release built for
+          # amd64 because they are architecture independent.
+          if [ "${ARCHITECTURE}" = "amd64" ]; then
+            if [ "${APT_TASK_NAMESPACE}" = "apt" ]; then
+              # Example: debian-bookworm-amd64 -> debian-bookworm
+              code_name="${ID%-*}"
+              # Example: debian-bookworm -> bookworm
+              code_name="${code_name#*-}"
+              # Create
+              # https://packages.apache.org/artifactory/arrow/${DISTRIBUTION}/apache-arrow-apt-source-latest-${code_name}.deb
+              # for easy to install.
+              cp -a \
+                "${DISTRIBUTION}/pool/${code_name}/a/apache-arrow-apt-source/*.deb \
+                "${DISTRIBUTION}/apache-arrow-apt-source-latest-${code_name}.deb"
+            else
+              # Example: amazon-linux-2023-amd64 -> amazon-linux-2023
+              version="${ID%-*}"
+              # Example: amazon-linux-2023 -> 2023
+              version="${version##*-}"
+              # Create
+              # https://packages.apache.org/artifactory/arrow/${DISTRIBUTION}/${version}/apache-arrow-release-latest.rpm
+              # for easy to install.
+              cp -a \
+                "${DISTRIBUTION}/${version}/x86_64/Packages/apache-arrow-release-*.rpm \
+                "${DISTRIBUTION}/${version}/apache-arrow-release-latest.rpm"
+            fi
+          fi


This is moved from binary-task.rb.

We'll create latest .deb/.rpm here to simplify binary-task.rb.

kou · 2025-10-02T07:49:36Z

.github/workflows/package_linux.yml

+          if [ "${GITHUB_REF_TYPE}" = "tag" ]; then
+            # Example: apache-arrow-21.0.0-rc0 -> 21.0.0-rc0
+            version="${GITHUB_REF_NAME#apache-arrow-}"
+            echo "ARROW_VERSION=${version}" >> "${GITHUB_ENV}"


We can use 21.0.0-rc0 not 21.0.0 here. The -rc0 part will be removed by

arrow/dev/tasks/linux-packages/package-task.rb

Lines 37 to 45 in 16ceade

when /-((dev|rc)\d+)\z/

base_version = $PREMATCH

sub_version = $1

type = $2

if type == "rc" and options[:rc_build_type] == :release

@deb_upstream_version = base_version

@deb_archive_base_name_version = base_version

@rpm_version = base_version

@rpm_release = "1"

.

github-actions bot added the awaiting committer review Awaiting committer review label Sep 19, 2025

raulcd commented Sep 19, 2025

View reviewed changes

.github/workflows/package_linux.yml Outdated Show resolved Hide resolved

dev/release/04-binary-download.sh Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Sep 19, 2025

raulcd commented Sep 19, 2025

View reviewed changes

github-actions bot added awaiting change review Awaiting change review awaiting changes Awaiting changes and removed awaiting changes Awaiting changes awaiting change review Awaiting change review labels Sep 19, 2025

raulcd force-pushed the GH-47582 branch from 9e3e8e4 to 66df8cb Compare September 19, 2025 11:30

kou reviewed Sep 19, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 19, 2025

raulcd force-pushed the GH-47582 branch 3 times, most recently from 1121d15 to 78e0ff2 Compare September 22, 2025 10:35

raulcd commented Sep 23, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes awaiting change review Awaiting change review and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 23, 2025

kou reviewed Sep 23, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review awaiting changes Awaiting changes labels Sep 23, 2025

raulcd and others added 14 commits October 2, 2025 16:11

Simplify apt:rc:copy and yum:rc:copy to maintain repo folder structur…

df7d3cb

…e from tar.gz

Remove old Linux Package tasks

f922af1

Uncomment the rest of linux package tasks

c803e8e

Remove missing tasks

168c696

Simplify .tar.gz folder structure more to be able to simplify even mo…

849bbba

…re binary tasks

Fix verify-yum.sh and verify-apt.sh to remove apt/repositories and yu…

0d8110b

…m/repositories subfolders Try fixing yum tests

Try to fix after understanding the code a bit better :)

14541a6

Revert "Try to fix after understanding the code a bit better :)"

5b6d51c

This reverts commit c9c4631.

Revert "Fix verify-yum.sh and verify-apt.sh to remove apt/repositorie…

2850339

…s and yum/repositories subfolders" This reverts commit 516e0fb.

Revert "Simplify .tar.gz folder structure more to be able to simplify…

36ea65b

… even more binary tasks" This reverts commit 471e110.

Some fixes to ruby as per code review

6c454d0

Apply suggestion from @kou

70bf6c6

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

Fix indent

f65170a

Simplify

f0088d4

* Use GitHub Release instead of GitHub Actions' artifacts * Build directory structure in package generator not binary-task.rb

kou force-pushed the GH-47582 branch from c7f2c59 to f0088d4 Compare October 2, 2025 07:11

Revert needless changes

3e27192

kou mentioned this pull request Oct 2, 2025

Follow Linux packages upload changes in apache/arrow apache/arrow-adbc#3510

Open

kou added 3 commits October 2, 2025 16:58

Remove needless changes

840607e

Not use SRPM

48edc55

Need free space

7673c04

kou reviewed Oct 2, 2025

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 2, 2025

Add support for arm64 image

a84b42a

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 2, 2025

kou added 4 commits October 2, 2025 18:06

Debug

bc1233d

Add support for arm64 image

b23f06f

Fix distribution and version split

738605c

Fix package version

d30e535

		# apache/arrow-adbc uses debian-bookworm. So the following
		# glob must much both of them.

-          tar cvzf ${{ matrix.id }}.tar.gz */${TASK_NAMESPACE}/repositories
+          mkdir -p artifacts
+          cp -a */${TASK_NAMESPACE}/repositories/* artifacts/
+          tar cvzf ${{ matrix.id }}.tar.gz -C artifacts .

	. "${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"
	"${SOURCE_DIR}/utils-watch-gh-workflow.sh" "${release_tag}" "package_linux.yml"

	when /-((dev\|rc)\d+)\z/
	base_version = $PREMATCH
	sub_version = $1
	type = $2
	if type == "rc" and options[:rc_build_type] == :release
	@deb_upstream_version = base_version
	@deb_archive_base_name_version = base_version
	@rpm_version = base_version
	@rpm_release = "1"

GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

Are you sure you want to change the base?

GH-47582: [CI][Packaging] Move linux-packaging tasks to apache/arrow repository #47600

Conversation

raulcd commented Sep 19, 2025 • edited by github-actions bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TBD

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

github-actions bot commented Sep 19, 2025

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

raulcd Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

raulcd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kou left a comment

Choose a reason for hiding this comment

raulcd commented Sep 19, 2025 •

edited by github-actions bot

Loading

raulcd Sep 19, 2025 •

edited

Loading