Skip to content

chore: update target-cpus in published binaries to x86-64-v3 and neoverse-n1 #3368

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mbutrovich wants to merge 5 commits into
base: main
Choose a base branch
from
+15 −9
Open

chore: update target-cpus in published binaries to x86-64-v3 and neoverse-n1 #3368

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
34f26f7
Adjust target-cpus to modern baselines. Remove release-linux flag for…
mbutrovich Feb 2, 2026
a66cedc
Undo inadvertent .PHONY target change.
mbutrovich Feb 2, 2026
4c3ac2f
Remove -prefer-256-bit flag.
mbutrovich Feb 3, 2026
190f951
Merge branch 'main' into update_cpu_targets
mbutrovich Feb 3, 2026
361e567
Add docs.
mbutrovich Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,26 +51,26 @@ format:

# build native libs for amd64 architecture Linux/MacOS on a Linux/amd64 machine/container
core-amd64-libs:
cd native && cargo build -j 2 --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo build -j 2 --release $(FEATURES_ARG)
Copy link
Contributor Author

@mbutrovich mbutrovich Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a target for released binaries (build-release-comet.sh).

ifdef HAS_OSXCROSS
rustup target add x86_64-apple-darwin
cd native && cargo build -j 2 --target x86_64-apple-darwin --release $(FEATURES_ARG)
endif

# build native libs for arm64 architecture Linux/MacOS on a Linux/arm64 machine/container
core-arm64-libs:
cd native && cargo build -j 2 --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=neoverse-n1" cargo build -j 2 --release $(FEATURES_ARG)
Copy link
Contributor Author

@mbutrovich mbutrovich Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is also a target for released binaries (build-release-comet.sh).

ifdef HAS_OSXCROSS
rustup target add aarch64-apple-darwin
cd native && cargo build -j 2 --target aarch64-apple-darwin --release $(FEATURES_ARG)
endif

core-amd64:
rustup target add x86_64-apple-darwin
cd native && RUSTFLAGS="-Ctarget-cpu=skylake -Ctarget-feature=-prefer-256-bit" CC=o64-clang CXX=o64-clang++ cargo build --target x86_64-apple-darwin --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=skylake" CC=o64-clang CXX=o64-clang++ cargo build --target x86_64-apple-darwin --release $(FEATURES_ARG)
Copy link
Contributor Author

@mbutrovich mbutrovich Feb 3, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flag doesn't make sense on this target (Skylake), so I suspect it's mistakenly copy-pasted from elsewhere.

mkdir -p common/target/classes/org/apache/comet/darwin/x86_64
cp native/target/x86_64-apple-darwin/release/libcomet.dylib common/target/classes/org/apache/comet/darwin/x86_64
cd native && RUSTFLAGS="-Ctarget-cpu=haswell -Ctarget-feature=-prefer-256-bit" cargo build --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=x86-64-v3" cargo build --release $(FEATURES_ARG)
Copy link
Contributor Author

@mbutrovich mbutrovich Feb 3, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for local development I guess, but made consistent. Also this flag doesn't make sense on this target (Haswell), so I suspect it's mistakenly copy-pasted from elsewhere.

mkdir -p common/target/classes/org/apache/comet/linux/amd64
cp native/target/release/libcomet.so common/target/classes/org/apache/comet/linux/amd64
jar -cf common/target/comet-native-x86_64.jar \
Expand All @@ -83,7 +83,7 @@ core-arm64:
cd native && RUSTFLAGS="-Ctarget-cpu=apple-m1" CC=arm64-apple-darwin21.4-clang CXX=arm64-apple-darwin21.4-clang++ CARGO_FEATURE_NEON=1 cargo build --target aarch64-apple-darwin --release $(FEATURES_ARG)
mkdir -p common/target/classes/org/apache/comet/darwin/aarch64
cp native/target/aarch64-apple-darwin/release/libcomet.dylib common/target/classes/org/apache/comet/darwin/aarch64
cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo build --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=neoverse-n1" cargo build --release $(FEATURES_ARG)
Copy link
Contributor Author

@mbutrovich mbutrovich Feb 3, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is for local development I guess, but made consistent.

mkdir -p common/target/classes/org/apache/comet/linux/aarch64
cp native/target/release/libcomet.so common/target/classes/org/apache/comet/linux/aarch64
jar -cf common/target/comet-native-aarch64.jar \
Expand All @@ -94,8 +94,8 @@ core-arm64:
release-linux: clean
rustup target add aarch64-apple-darwin x86_64-apple-darwin
cd native && RUSTFLAGS="-Ctarget-cpu=apple-m1" CC=arm64-apple-darwin21.4-clang CXX=arm64-apple-darwin21.4-clang++ CARGO_FEATURE_NEON=1 cargo build --target aarch64-apple-darwin --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=skylake -Ctarget-feature=-prefer-256-bit" CC=o64-clang CXX=o64-clang++ cargo build --target x86_64-apple-darwin --release $(FEATURES_ARG)
Copy link
Contributor Author

@mbutrovich mbutrovich Feb 3, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This flag doesn't make sense on this target (Skylake), so I suspect it's mistakenly copy-pasted from elsewhere.

cd native && RUSTFLAGS="-Ctarget-cpu=native -Ctarget-feature=-prefer-256-bit" cargo build --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=skylake" CC=o64-clang CXX=o64-clang++ cargo build --target x86_64-apple-darwin --release $(FEATURES_ARG)
cd native && RUSTFLAGS="-Ctarget-cpu=native" cargo build --release $(FEATURES_ARG)
./mvnw install -Prelease -DskipTests $(PROFILES)
release:
cd native && RUSTFLAGS="$(RUSTFLAGS) -Ctarget-cpu=native" cargo build --release $(FEATURES_ARG)
Expand Down
10 changes: 8 additions & 2 deletions docs/source/user-guide/latest/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ Cloud Service Providers.
Comet jar files are available in [Maven Central](https://central.sonatype.com/namespace/org.apache.datafusion) for amd64 and arm64 architectures for Linux. For Apple macOS, it
is currently necessary to build from source.

For performance reasons, published Comet jar files target baseline CPUs available in modern data centers. For example,
the amd64 build uses the `x86-64-v3` target that adds CPU instructions (_e.g._, AVX2) common after 2013. Similarly, the
arm64 build uses the `neoverse-n1` target, which is a common baseline for ARM cores found in AWS (Graviton2+), GCP, and
Azure after 2019. If the Comet library fails for SIGILL (illegal instruction), please open an issue on the GitHub
repository describing your environment, and [build from source] for your target architecture.

Here are the direct links for downloading the Comet $COMET_VERSION jar file. Note that these links are not valid if
you are viewing the documentation for the latest `SNAPSHOT` release, or for the latest release while it is still going
through the release vote. Release candidate jars can be found [here](https://repository.apache.org/#nexus-search;quick~org.apache.datafusion).
Expand All @@ -72,10 +78,10 @@ through the release vote. Release candidate jars can be found [here](https://rep

## Building from source

Refer to the [Building from Source] guide for instructions from building Comet from source, either from official
Refer to the [Building from source] guide for instructions from building Comet from source, either from official
source releases, or from the latest code in the GitHub repository.

[Building from Source]: source.md
[Building from source]: source.md

## Deploying to Kubernetes

Expand Down
Loading