Provide pre-built wheels

[marcm-ml]

Feature Request / Improvement

Hello,

I would like to have prebuilt wheels so there is no need for compilation when installing pyiceberg. There is a considerable installation speed penalty (~8x slower) when installing pyiceberg in docker images when using QEMU emulation (building arm64 on amd64 machines). Providing prebuilt wheels would not have this issue.

[rambleraptor]

PyPi lists wheels for multiple OS/architecture combinations. Is there a different one that you'd like us to support?

[kevinjqliu]

hey! we have prebuilt wheels as part of the release, https://pypi.org/project/pyiceberg/0.10.0/#files
it might be that we are not building the specific one that you need so the fallback is to rebuild from source.

Do you have more info on the specific architecture that youre looking for?

[rambleraptor]

Using Pyarrow as a comparison, it looks like they publish musl aarch64, where we don't (i.e. musllinux_1_2_aarch64). I'll look into how hard it would be to publish those as well.

[raulcd]

@rambleraptor from what I can see manylinux aarch64 also seem to not be provided in case you want to take a loot at those too. I generally release the wheels for pyarrow, let me know if I can help.

[kevinjqliu]

ooo thanks for the pointer @raulcd. We were talking about this during todays pyiceberg sync and was wondering what pyarrow did.
heres how we build the wheels

iceberg-python/.github/workflows/svn-build-artifacts.yml

Lines 59 to 74 in fe6b6fc

	- name: Build wheels
	uses: pypa/cibuildwheel@v3.3.1
	with:
	output-dir: wheelhouse
	config-file: "pyproject.toml"
	env:
	# Ignore 32 bit architectures
	CIBW_ARCHS: "auto64"
	CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.14"
	CIBW_BEFORE_TEST: "uv sync --directory {project} --only-group dev --no-install-project"
	CIBW_TEST_COMMAND: "uv run --directory {project} pytest tests/avro/test_decoder.py"
	# Ignore tests for pypy since not all dependencies are compiled for it
	# and would require a local rust build chain
	CIBW_TEST_SKIP: "pp*"
	# Skip free-threaded (PEP 703) builds until we evaluate decoder_fast support
	CIBW_SKIP: "cp3*t-*"

iceberg-python/.github/workflows/pypi-build-artifacts.yml

Lines 64 to 79 in fe6b6fc

	- name: Build wheels
	uses: pypa/cibuildwheel@v3.3.1
	with:
	output-dir: wheelhouse
	config-file: "pyproject.toml"
	env:
	# Ignore 32 bit architectures
	CIBW_ARCHS: "auto64"
	CIBW_PROJECT_REQUIRES_PYTHON: ">=3.10,<3.14"
	CIBW_BEFORE_TEST: "uv sync --directory {project} --only-group dev --no-install-project"
	CIBW_TEST_COMMAND: "uv run --directory {project} pytest tests/avro/test_decoder.py"
	# Ignore tests for pypy since not all dependencies are compiled for it
	# and would require a local rust build chain
	CIBW_TEST_SKIP: "pp*"
	# Skip free-threaded (PEP 703) builds until we evaluate decoder_fast support
	CIBW_SKIP: "cp3*t-*"

[marcm-ml]

Hey, thanks for the replies. I might have found the reason, we are using python 3.13 and as far as I can see, there are only wheels for 3.9-3.12.

Also the specific platform is Linux arm64

[kevinjqliu]

upcoming release (0.11) should have 3.13 support 🔥 we're working on finish up the release
its already part of the nightly build https://test.pypi.org/project/pyiceberg/#files

[marcm-ml]

Perfect, that’s good to know. If you like, you can close this issue if you prefer to have the discussion around aarch in a separate one.

[marcm-ml]

Nevermind, I cannot see arm builds for the nightly manylinux wheel. Are you sure that you provide these?

Edit: I suppose for manylinux builds this is called aarch64. I got confused by that, so raulcd comment also points out the same missing wheels.

[raulcd]

We don't use cibuildwheel on arrow but from my understanding of CIBW_ARCHS: "auto64" it only builds the architecture supported on the machine. If you want to use qemu you have to specifically select the CIBW_ARCHS_LINUX: "auto aarch64" .
https://github.com/pypa/cibuildwheel/blob/7989c25bcd7060e0fc691279f41643a4b9c4cf1c/docs/options.md#archs--archs-cmd-line-env-var-toml-

Another possible approach would be to use the same setup but add arm runners for the aarch64 wheels.

[rambleraptor]

I just sent up a PR. I would love some reviews on it!